AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

Pengyu He, Yan Zhang, Ruien Li +1

The pre-training of Large Language Models (LLMs) is increasingly conducted across multiple data centers. As training scales to a larger number of accelerators, the fraction of time…