Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts

Tho Tran Huu, Huu-Tuan Nguyen, Thien-Hai Nguyen +4

Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networ…