4 papers
SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
Qifan Yu, Xinyu Ma, Zhijian Zhuo +7
Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansio…
Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better
Ji Zhao, Yufei Gu, Shitong Shao +3
As Large Language Models (LLMs) achieve remarkable empirical success through scaling model and data size, pretraining has become increasingly critical yet computationally prohibiti…
Robust LLM Training Infrastructure at ByteDance
Borui Wan, Gaohong Liu, Zuquan Song +32
The training scale of large language models (LLMs) has reached tens of thousands of GPUs and is still continuously expanding, enabling faster learning of larger models. Accompanyin…
Balanced Actor Initialization: Stable RLHF Training of Distillation-Based Reasoning Models
Chen Zheng, Yiyuan Ma, Yuan Yang +11
The development of alignment and reasoning capabilities in large language models has seen remarkable progress through two paradigms: instruction tuning and reinforcement learning f…