MoRA: MoE Pruning via Router Bias Learning and Expert Approximation

Yushuai Sun, Zikun Zhou, Lin Gao +2

Mixture-of-Experts (MoE) models enable parameter scaling with limited per-token computation by activating only a small subset of experts for each token, but deploying them still re…