Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

Kirill Labzin, Stepan Kulibaba, Artem Dzhalilov +1

Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer to select experts for every token. Prior work has shown that routing decisions…