Data movement limits to frontier model training

Ege Erdil, David Schneider-Joseph

We present a theoretical model of distributed training, and use it to analyze how far dense and sparse training runs can be scaled. Under our baseline assumptions, given a three mo…