4 papers · 1 filter
GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training
Jieying Wang, Shuyuan Fan, Zizhong Wang +4
Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can signi…
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Haotian Xie, Junlin Chen, Mingkai Zheng +2
State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware…
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Mingkai Zheng, Junlin Chen, Haotian Xie +1
Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronizatio…
Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safety
Can Jin, Rui Wu, Tong Che +10
Ensuring that Large Language Models (LLMs) adhere to safety principles without refusing benign requests remains a significant challenge. While OpenAI introduces deliberative alignm…