AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models

Nikolay Kutuzov, Makar Baderko, Stepan Kulibaba +4

Scaling distributed training of Large Language Models (LLMs) requires not only algorithmic advances but also efficient utilization of heterogeneous hardware resources. While existi…