9 papers
GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training
Jieying Wang, Shuyuan Fan, Zizhong Wang +4
Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can signi…
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Haotian Xie, Junlin Chen, Mingkai Zheng +2
State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware…
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Mingkai Zheng, Junlin Chen, Haotian Xie +1
Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronizatio…
Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safety
Can Jin, Rui Wu, Tong Che +10
Ensuring that Large Language Models (LLMs) adhere to safety principles without refusing benign requests remains a significant challenge. While OpenAI introduces deliberative alignm…
Pier: Efficient Large Language Model pretraining with Relaxed Global Communication
Shuyuan Fan, Zhao Zhang
Global communication, such as all-reduce and allgather, is the prominent performance bottleneck in large language model (LLM) pretraining. To address this issue, we present Pier, a…
When Harmless Words Harm: A New Threat to LLM Safety via Conceptual Triggers
Zhaoxin Zhang, Borui Chen, Yiming Hu +3
Recent research on large language model (LLM) jailbreaks has primarily focused on techniques that bypass safety mechanisms to elicit overtly harmful outputs. However, such efforts…
Maple: A Multi-agent System for Portable Deep Learning across Clusters
Molang Wu, Zhao Zhang
Training deep learning (DL) models across Graphics Processing Unit (GPU) clusters is technically challenging. One aspect is that users have to compose command lines to adapt to the…
Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM
Biyao Zhang, Mingkai Zheng, Debargha Ganguly +4
Training Large Language Models(LLMs) is one of the most compute-intensive tasks in high-performance computing. Predicting end-to-end training time for multi-billion parameter model…
Aligning Teacher with Student Preferences for Tailored Training Data Generation
Yantao Liu, Zhao Zhang, Zijun Yao +3
Large Language Models (LLMs) have shown significant promise as copilots in various tasks. Local deployment of LLMs on edge devices is necessary when handling privacy-sensitive data…