4 papers
GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining
Jieying Wang, Shuyuan Fan, Mingkai Zheng +1
Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can signi…
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Haotian Xie, Junlin Chen, Mingkai Zheng +2
State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware…
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Mingkai Zheng, Junlin Chen, Haotian Xie +1
Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronizatio…
Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM
Biyao Zhang, Mingkai Zheng, Debargha Ganguly +4
Training Large Language Models(LLMs) is one of the most compute-intensive tasks in high-performance computing. Predicting end-to-end training time for multi-billion parameter model…