most citedMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

24 citations · 25 across the 4 of their papers we have counts for

collaborators
Showing cs.LGShow all

2 papers · 1 filter

cs.LG2026

Lossless Anti-Distillation Sampling

Zibo Diao, Jingchu Gai, Xinyue Ai +3

Frontier commercial generative models face a growing threat from distillation, whereby a distiller harvests generated responses and trains a competing model at drastically lower co…

cs.LG2024★ 24 cited

MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

Ziheng Jiang, Haibin Lin, Yinmin Zhong +29

We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of…