Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

Xing Chen, Hengshuai Yao

Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization impl…