Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures

Minh-Khoi Nguyen-Nhat, Rachel S. Y. Teo, Laziz Abdullaev +3

Sparse Mixture of Experts (SMoE) has emerged as a promising solution to achieving unparalleled scalability in deep learning by decoupling model parameter count from computational c…