Selective Sinkhorn Routing for Improved Sparse Mixture of Experts

Duc Anh Nguyen, Huu Binh Ta, Nhuan Le Duc +2

Sparse Mixture-of-Experts (SMoE) models are scalable and computationally efficient, enabling large increases in model capacity with limited inference overhead. Existing SMoE method…