#mixture-of-experts
9 papers · 1 filter
Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers
Chongjian Ge, Hanwen Jiang, Tianyu Wang +9
The paper presents Chimera, a hybrid visual diffusion transformer that processes text, image, and video tokens in a single raster-ordered stream using efficient attention mechanism…
Heterogeneous Ranking in Industrial-Scale Recommender Systems: A Case Study
Di Bai, Jintao Liu, Zhenwei Tang +3
The paper describes an industrial case study of ranking heterogeneous content feeds in Google Discover using a heterogeneity-adaptive multi-gated mixture-of-experts model (HA-MoE)…
Route-Block Membership Selects Packed-AWQ Arithmetic: A Controlled Single-Fixture Mechanism Study
Lukas Stepanek
The paper studies how the alignment of routed tokens into expert blocks determines the exact packed quantized matrix multiplication performed during Mixture‑of‑Experts inference, d…
Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA
Tom Saliencro, Rohan Desai, Priya Nair +2
The paper introduces CARE, a confidence-adaptive routing method for Mixture-of-Experts LoRA that dynamically selects the number of experts per token based on the router’s uncertain…
Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit
Xiaomi MiMo Team, Anqi Liu, Aoxin Ma +28
The paper describes a production-ready inference system for the MiMo-V2.5 large language model family that combines hybrid sliding window attention, sparse mixture-of-experts, and…
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jincheng Xie, Runheng Liu, Heyan Huang +4
The paper introduces EcoSpec, a cost-aware speculative decoding method that selects draft tokens to minimize expert activation overhead in large mixture-of-experts language models,…