MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation

Lie Li, Wen Li, Junxiao Shen +2

Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heteroge…