3 papers
cs.LG2026
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Jiapeng Wang, Changxin Tian, Kunlong Chen +5
Optimizing data mixtures is essential for unlocking the full potential of large language models (LLMs), yet identifying the optimal composition remains computationally prohibitive…
cs.CL2025
Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data
Jiapeng Wang, Yiwen Hu, Yanzipeng Gao +7
As access to high-quality, domain-specific data grows increasingly scarce, multi-epoch training has become a practical strategy for adapting large language models (LLMs). However,…
cs.CL2025
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Changxin Tian, Jiapeng Wang, Qian Zhao +7
Recent advances in learning rate (LR) scheduling have demonstrated the effectiveness of decay-free approaches that eliminate the traditional decay phase while maintaining competiti…