DiLoCo: Distributed Low-Communication Training of Language Models

Arthur Douillard, Qixuan Feng, Andrei A. Rusu +6

Large language models (LLM) have become a critical component in many applications of machine learning. However, standard approaches to training LLM require a large number of tightl…