Showing cs.DCShow all
2 papers · 1 filter
cs.DC2024
Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
Jiangfei Duan, Shuo Zhang, Zerui Wang +13
Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and sign…
cs.DC2024
AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training
Qiaoling Chen, Qinghao Hu, Guoteng Wang +8
Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer…