4 papers · 1 filter
GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training
Jieying Wang, Shuyuan Fan, Zizhong Wang +4
Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can signi…
Pier: Efficient Large Language Model pretraining with Relaxed Global Communication
Shuyuan Fan, Zhao Zhang
Global communication, such as all-reduce and allgather, is the prominent performance bottleneck in large language model (LLM) pretraining. To address this issue, we present Pier, a…
Maple: A Multi-agent System for Portable Deep Learning across Clusters
Molang Wu, Zhao Zhang
Training deep learning (DL) models across Graphics Processing Unit (GPU) clusters is technically challenging. One aspect is that users have to compose command lines to adapt to the…
Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM
Biyao Zhang, Mingkai Zheng, Debargha Ganguly +4
Training Large Language Models(LLMs) is one of the most compute-intensive tasks in high-performance computing. Predicting end-to-end training time for multi-billion parameter model…