An Empirical Model of Large-Batch Training

Sam McCandlish, Jared Kaplan, Dario Amodei +1

In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However…