Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

Jiahe Fan, Yinghao Hou, Si Chen +3

Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment? Existing heterogeneous fusio…