Collective Communication for 100k+ GPUs

Min Si, Pavan Balaji, Yongzhou Chen +36

The increasing scale of large language models (LLMs) necessitates highly efficient collective communication frameworks, particularly as training workloads extend to hundreds of tho…