Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

Jia-Nan Wang, Zixun Huang, Kairui Li +1

We study when and how momentum improves large-batch training in the one-pass regime, using power-law kernel regression as a tractable setting. We first characterize risk stability…