Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

Jared Fernandez, Luca Wehrstedt, Leonid Shamis +5

Dramatic increases in the capabilities of neural network models in recent years are driven by scaling model size, training data, and corresponding computational resources. To devel…