Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer

Boan Liu, Liang Ding, Li Shen +4

The Mixture of Experts (MoE) has emerged as a highly successful technique in deep learning, based on the principle of divide-and-conquer to maximize model capacity without signific…