5 papers · 1 filter
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7
Pre-training large language models on massive GPU clusters has made hardware faults routine rather than rare, driving the need for resilient training systems. Yet existing framewor…
Implementing True MPI Sessions and Evaluating MPI Initialization Scalability
Hui Zhou, Kenneth Raffenetti, Yanfei Guo +2
Sessions is one of the major features introduced in the MPI-4 standard. It offers an alternative to the traditional world communicator model by allowing applications to construct c…
ZCCL: Significantly Improving Collective Communication With Error-Bounded Lossy Compression
Jiajun Huang, Sheng Di, Xiaodong Yu +12
With the ever-increasing computing power of supercomputers and the growing scale of scientific applications, the efficiency of MPI collective communication turns out to be a critic…
MPI Progress For All
Hui Zhou, Robert Latham, Ken Raffenetti +2
The progression of communication in the Message Passing Interface (MPI) is not well defined, yet it is critical for application performance, particularly in achieving effective com…
gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusters
Jiajun Huang, Sheng Di, Xiaodong Yu +11
GPU-aware collective communication has become a major bottleneck for modern computing platforms as GPU computing power rapidly rises. A traditional approach is to directly integrat…