Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques

Shwai He, Daize Dong, Liang Ding +1

Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world dep…