24 citations · 25 across the 3 of their papers we have counts for
Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training
Xinwei Qiang, Hongmin Chen, Shixuan Sun +3
Determinism is indispensable for reproducibility in large language model (LLM) training, yet it often exacts a steep performance cost. In widely used attention implementations such…
cs.LG2024★ 24 cited
MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
Ziheng Jiang, Haibin Lin, Yinmin Zhong +29
We present the design, implementation and engineering experience in building and deploying MegaScale, a production system for training large language models (LLMs) at the scale of…