LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing

Jiawei Hao, Zhiwei Hao, Jianyuan Guo +4

Mixture-of-Experts (MoE) based Large Language Models (LLMs) have demonstrated impressive performance and computational efficiency. However, their deployment is often constrained by…