The power of fine-grained experts: Granularity boosts expressivity in Mixture of Experts

Enric Boix-Adsera, Philippe Rigollet

Mixture-of-Experts (MoE) layers are increasingly central to frontier model architectures. By selectively activating parameters, they reduce computational cost while scaling total p…