4 papers
MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate
Jianze Wang, Ying Liu, Jinlong Chen +7
On-policy distillation (OPD) trains a student on its own trajectories under token-level teacher supervision, but existing methods are capped by a single-teacher capability ceiling:…
GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving
Shakya Jayakody, Youpeng Zhao, Chinmay Dhanraj Nehate +1
The rise of million-token, agent-based applications has placed unprecedented demands on large language model (LLM) inference services. The long-running nature of these tasks increa…
Robust LLM Training Infrastructure at ByteDance
Borui Wan, Gaohong Liu, Zuquan Song +32
The training scale of large language models (LLMs) has reached tens of thousands of GPUs and is still continuously expanding, enabling faster learning of larger models. Accompanyin…
Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs
Guoliang He, Youhe Jiang, Wencong Xiao +8
The scaling law for large language models (LLMs) depicts that the path towards machine intelligence necessitates training at large scale. Thus, companies continuously build large-s…