MergeMoE: Efficient Compression of MoE Models via Expert Output Merging

Ruijie Miao, Yilun Yao, Zihan Wang +5

The Mixture-of-Experts (MoE) technique has proven to be a promising solution to efficiently scale the model size, which has been widely applied in recent LLM advancements. However,…