Rethinking Routing Exploration in Modern Mixture-of-Experts

Bingcong Li, Hao Ma, Michael Muehlebach +2

Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained exper…