Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement

Le Yu, Bowen Yu, Haiyang Yu +2

Merging Large Language Models (LLMs) aims to amalgamate multiple homologous LLMs into one with all the capabilities. Ideally, any LLMs sharing the same backbone should be mergeable…