Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Huiyuan Tian, Bonan Xu, Shijian Li

Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinc…