Model Merging in Pre-training of Large Language Models

Yunshui Li, Yiyuan Ma, Shen Yan +23

Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this pa…