11 papers
Hyperparameter Scaling Laws Across MoE Sparsity
Changxin Tian, Kunlong Chen, Jia Liu +3
Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challengin…
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
Qian Zhao, Kunlong Chen, Changxin Tian +9
FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class…
SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks
Fake Lin, Binbin Hu, Xi Zhu +6
Code Agents have achieved remarkable advances in recent years, exhibiting strong capabilities across a wide range of software engineering tasks. However, their misuse often produce…
Token-level Collaborative Alignment for LLM-based Generative Recommendation
Fake Lin, Binbin Hu, Zhi Zheng +5
Large Language Models (LLMs) have demonstrated strong potential for generative recommendation by leveraging rich semantic knowledge. However, existing LLM-based recommender systems…
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Jiapeng Wang, Changxin Tian, Kunlong Chen +5
Optimizing data mixtures is essential for unlocking the full potential of large language models (LLMs), yet identifying the optimal composition remains computationally prohibitive…
Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation
Ling Team, Ang Li, Ben Liu +138
We introduce Ling 2.0, a series reasoning-oriented language foundation built upon the principle that every activation boosts reasoning capability. Designed to scale from tens of bi…
MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics
Jiapeng Wang, Changxin Tian, Kunlong Chen +5
Reliable evaluation is fundamental to the progress of Large Language Models (LLMs), yet the evaluation process during pre-training is plagued by significant instability that obscur…
CodeChemist: Test-Time Scaling for Low-Resource Code Generation via Functional Knowledge Transfer
Kaixin Wang, Tianlin Li, Xiaoyu Zhang +6
Code Large Language Models (CodeLLMs) have been widely adopted for Natural Language to Programming Language code generation, powering applications with large user bases. Their perf…
Arrows of Math Reasoning Data Synthesis for Large Language Models: Diversity, Complexity and Correctness
Sirui Chen, Changxin Tian, Binbin Hu +4
Enhancing the mathematical reasoning of large language models (LLMs) demands high-quality training data, yet conventional methods face critical challenges in scalability, cost, and…
WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training
Changxin Tian, Jiapeng Wang, Qian Zhao +7
Recent advances in learning rate (LR) scheduling have demonstrated the effectiveness of decay-free approaches that eliminate the traditional decay phase while maintaining competiti…
Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
Changxin Tian, Kunlong Chen, Jia Liu +3
Mixture-of-Experts (MoE) has become a dominant architecture for scaling Large Language Models (LLMs) efficiently by decoupling total parameters from computational cost. However, th…