Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference

Sneha Kudugunta, Yanping Huang, Ankur Bapna +4

Sparse Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training com…