Retraining-Free Merging of Sparse MoE via Hierarchical Clustering

I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun +3

Sparse Mixture-of-Experts (SMoE) models represent a significant advancement in large language model (LLM) development through their efficient parameter utilization. These models ac…