Heterogeneous Low-Bandwidth Pre-Training of LLMs

Yazan Obeidi, Amir Sarfi, Joel Lidin +2

Pre-training large language models (LLMs) increasingly requires distributed compute, yet bandwidth constraints make it difficult to scale beyond well-provisioned datacenters-especi…