Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression

Tianxiao Cao, Jiahe Shao, Yuning Qiu +3

Mixture-of-Experts (MoE) large language models decouple capacity from compute through sparse routing, but their large parameter count creates storage and serving challenges. We ana…