Large Batch Training of Convolutional Networks

Abstract

A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.

References in corpus (4)

• One weird trick for parallelizing convolutional neural networks
• Revisiting Distributed Synchronous SGD
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Diagonal Rescaling For Neural Networks

Cited by in corpus (9)

• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
• Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search
• Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training
• Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD
• Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks
• Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning
• Convergence Analysis of Gradient Descent Algorithms with Proportional Updates
• Slim-DP: A Light Communication Data Parallelism for DNN