DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster

Ji Qi, WenPeng Zhu, Li Li +6

The distributed training of foundation models, particularly large language models (LLMs), demands a high level of communication. Consequently, it is highly dependent on a centraliz…