6 papers
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
Yuanyi Wang, Yanggan Gu, Su Lu +5
Weight-space model merging is usually formulated as an algebraic operation on checkpoints, yet at LLM scale the limiting resource is often the set of expert weights that must be re…
MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging
Yuanyi Wang, Yanggan Gu, Zihao Wang +6
Large language model (LLM) merging has become a key technique in modern LLM development pipelines, enabling the integration of multiple task- or domain-specific expert models witho…
Model Merging Scaling Laws in Large Language Models
Yuanyi Wang, Yanggan Gu, Yiming Zhang +6
We study empirical scaling laws for language model merging measured by cross-entropy. Despite its wide practical use, merging lacks a quantitative rule that predicts returns as we…
InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion
Yuanyi Wang, Zhaoyi Yan, Yiming Zhang +4
Recent advances in large language models (LLMs) have intensified efforts to fuse heterogeneous open-source models into a unified system that inherits their complementary strengths.…
InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
Yanggan Gu, Yuanyi Wang, Zhaoyi Yan +4
Model fusion combines multiple Large Language Models (LLMs) with different strengths into a more powerful, integrated model through lightweight training methods. Existing works on…
InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion
Zhaoyi Yan, Yiming Zhang, Baoyi He +7
We introduce InfiFusion, an efficient training pipeline designed to integrate multiple domain-specialized Large Language Models (LLMs) into a single pivot model, effectively harnes…