MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging

Jiapeng Wang, Changxin Tian, Kunlong Chen +5

Optimizing data mixtures is essential for unlocking the full potential of large language models (LLMs), yet identifying the optimal composition remains computationally prohibitive…