2 papers
cs.DC2026
Training LLMs with Fault Tolerant HSDP on 100,000 GPUs
Omkar Salpekar, Rohan Varma, Kenny Yu +20
Large-scale training systems typically use synchronous training, requiring all GPUs to be healthy simultaneously. In our experience training on O(100K) GPUs, synchronous training r…
cs.DC2025
Collective Communication for 100k+ GPUs
Min Si, Pavan Balaji, Yongzhou Chen +36
The increasing scale of large language models (LLMs) necessitates highly efficient collective communication frameworks, particularly as training workloads extend to hundreds of tho…