Unifying Local Communications and Local Updates for LLM Pretraining

Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon

Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and lower-bandwidth links. Many pract…