Multi-Head Mixture-of-Experts

Xun Wu, Shaohan Huang, Wenhui Wang +1

Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activat…