Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training

Abstract

Stochastic gradient descent~(SGD) and its variants have been the dominating optimization methods in machine learning. Compared to SGD with small-batch training, SGD with large-batch training can better utilize the computational power of current multi-core systems such as graphics processing units~(GPUs) and can reduce the number of communication rounds in distributed training settings. Thus, SGD with large-batch training has attracted considerable attention. However, existing empirical results showed that large-batch training typically leads to a drop in generalization accuracy. Hence, how to guarantee the generalization ability in large-batch training becomes a challenging task. In this paper, we propose a simple yet effective method, called stochastic normalized gradient descent with momentum~(SNGM), for large-batch training. We prove that with the same number of gradient computations, SNGM can adopt a larger batch size than momentum SGD~(MSGD), which is one of the most widely used variants of SGD, to converge to an $\epsilon$-stationary point. Empirical results on deep learning verify that when adopting the same large batch size, SNGM can achieve better test accuracy than MSGD and other state-of-the-art large-batch training methods.

References in corpus (8)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• Towards Efficient and Scalable Sharpness-Aware Minimization