8 papers
Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining
Guanghao Zhu, Zeyu Liu, Zhitian Hou +10
Biomedical figures are explained not by captions alone but by body-text passages that discuss them. Yet current multimodal corpora typically reduce figures to isolated image-captio…
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
Yuanyi Wang, Yanggan Gu, Su Lu +5
Weight-space model merging is usually formulated as an algebraic operation on checkpoints, yet at LLM scale the limiting resource is often the set of expert weights that must be re…
Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
Yuanyi Wang, Su Lu, Yanggan Gu +6
On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by pr…
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
Zunhai Su, Rui Yang, Chao Zhang +11
The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficien…
E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring
Wenjun Wang, Yanggan Gu, Shuo Cai +4
Low-resource deployment constraints have made model quantization essential for deploying neural networks while preserving performance. Meanwhile, model merging has become an increa…
FeatCal: Feature Calibration for Post-Merging Models
Yanggan Gu, Shuo Cai, Zihao Wang +7
Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task exper…