MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs

Xiaodong Chen, Mingming Ha, Zhenzhong Lan +2

The Mixture-of-Experts (MoE) architecture has become a predominant paradigm for scaling large language models (LLMs). Despite offering strong performance and computational efficien…