MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

Yuhang Zhou, Giannis Karamanolakis, Victor Soto +5

The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert…