SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs

Mikołaj Zasada, Łukasz Struski, Jacek Tabor +1

Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-$k$ routing. While t…