Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation

Yaoxiang Wang, Kai Zhang, Ruobing Xie +4

Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing granularity. LatentMoE moves…