GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining

Jieying Wang, Shuyuan Fan, Mingkai Zheng +1

Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can signi…