Mixture of Group Experts for Learning Invariant Representations

Lei Kang, Jia Li, Mi Tian +1

Sparsely activated Mixture-of-Experts (MoE) models effectively increase the number of parameters while maintaining consistent computational costs per token. However, vanilla MoE mo…