Mixture of Neuron Experts

Runxi Cheng, Yuchen Guan, Yucheng Ding +6

In this work, we first explore whether the parameters activated by the MoE layer remain highly sparse at inference. We perform a sparsification study on several representative MoE…