11 papers
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
Yuanyi Wang, Yanggan Gu, Su Lu +5
Weight-space model merging is usually formulated as an algebraic operation on checkpoints, yet at LLM scale the limiting resource is often the set of expert weights that must be re…
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
Yuanyi Wang, Su Lu, Yanggan Gu +6
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by pr…
E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring
Wenjun Wang, Yanggan Gu, Shuo Cai +4
Low-resource deployment constraints have made model quantization essential for deploying neural networks while preserving performance. Meanwhile, model merging has become an increa…
Discovering Physical Directions in Weight Space: Composing Neural PDE Experts
Pengkai Wang, Pengwei Liu, Yuanyi Wang +7
Recent advances in neural operators have made partial differential equation (PDE) surrogate modeling increasingly scalable and transferable through large-scale pretraining and in-c…
FeatCal: Feature Calibration for Post-Merging Models
Yanggan Gu, Shuo Cai, Zihao Wang +7
Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task exper…
Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training
Yuanyi Wang, Yifan Yang, Su Lu +9
Continual post-training aims to extend large language models (LLMs) with new knowledge, skills, and behaviors, yet it remains unclear when sequential updates enable capability tran…
InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluation
Yifan Yang, Jinjia Li, Kunxi Li +7
The rapid advancement of large language models (LLMs) demands increasingly reliable evaluation, yet current centralized evaluation suffers from opacity, overfitting, and hardware-i…
MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging
Yuanyi Wang, Yanggan Gu, Zihao Wang +6
Large language model (LLM) merging has become a key technique in modern LLM development pipelines, enabling the integration of multiple task- or domain-specific expert models witho…
Model Merging Scaling Laws in Large Language Models
Yuanyi Wang, Yanggan Gu, Yiming Zhang +6
We study empirical scaling laws for language model merging measured by cross-entropy. Despite its wide practical use, merging lacks a quantitative rule that predicts returns as we…
InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion
Yuanyi Wang, Zhaoyi Yan, Yiming Zhang +4
Recent advances in large language models (LLMs) have intensified efforts to fuse heterogeneous open-source models into a unified system that inherits their complementary strengths.…
InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
Yanggan Gu, Yuanyi Wang, Zhaoyi Yan +4
Model fusion combines multiple Large Language Models (LLMs) with different strengths into a more powerful, integrated model through lightweight training methods. Existing works on…