Soft-to-Hard Routing in Sparse Mixture-of-Experts Models

Reza Rastegar

Softmax routing approaches hard top-1 routing as the temperature tends to zero, but the limiting passage is singular at router ties. This paper develops a boundary-layer calculus f…