Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery

Shengze Xu, Maolin Deng, Tieyong Zeng

Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others. We study how this concentration forms, why it per…