749 citations · 1.3k across the 36 of their papers we have counts for
4 papers · 2 filters
Scaling Laws for Looped Mixture of Experts
Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi
Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity e…
MobileMoE: Scaling On-Device Mixture of Experts
Yanbei Chen, Hanxian Huang, Ernie Chang +5
Mixture-of-Experts (MoE) has become the de facto architecture for hundred-billion-parameter language models, yet its advantages at sub-billion scales for on-device deployment remai…
WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Points
Dongyue Li, Zechun Liu, Kai Yi +6
Quantization-aware training (QAT) is widely adopted to quantize language models by training full-precision weights using gradients from the quantized model. The main bottleneck is…
MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment
Hanxian Huang, Igor Fedorov, Andrey Gromov +14
Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce nea…