Cross-region Model Training with Communication-Computation Overlapping and Delay Compensation

Ying Zhu, Yang Xu, Hongli Xu +3

Training large language models (LLMs) requires massive computational resources, often necessitating the aggregation of geographically distributed data centers (\ie, cross-region tr…