Large Batch Training Does Not Need Warmup
arXiv:2002.01576
Abstract
Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. However, the optimizer converges slowly at early epochs and there is a gap between large-batch deep learning optimization heuristics and theoretical underpinnings. In this paper, we propose a novel Complete Layer-wise Adaptive Rate Scaling (CLARS) algorithm for large-batch training. We also analyze the convergence rate of the proposed method by introducing a new fine-grained analysis of gradient-based methods. Based on our analysis, we bridge the gap and illustrate the theoretical insights for three popular large-batch training techniques, including linear learning rate scaling, gradual warmup, and layer-wise adaptive rate scaling. Extensive experiments demonstrate that the proposed algorithm outperforms gradual warmup technique by a large margin and defeats the convergence of the state-of-the-art large-batch optimizer in training advanced deep neural networks (ResNet, DenseNet, MobileNet) on ImageNet dataset.
References in corpus (6)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- One weird trick for parallelizing convolutional neural networks
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Large Batch Training of Convolutional Networks
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- Large-Batch Training for LSTM and Beyond