DirMoE: Dirichlet-routed Mixture of Experts

Amirhossein Vahidi, Hesam Asadollahzadeh, Navid Akhavan Attar +4

Mixture-of-Experts (MoE) models have demonstrated exceptional performance in large-scale language models. Existing routers typically rely on non-differentiable Top-$k$+Softmax, lim…