On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent

Abstract

Increasing the mini-batch size for stochastic gradient descent offers significant opportunities to reduce wall-clock training time, but there are a variety of theoretical and systems challenges that impede the widespread success of this technique. We investigate these issues, with an emphasis on time to convergence and total computational cost, through an extensive empirical analysis of network training across several architectures and problem domains, including image classification, image segmentation, and language modeling. Although it is common practice to increase the batch size in order to fully exploit available computational resources, we find a substantially more nuanced picture. Our main finding is that across a wide range of network architectures and problem domains, increasing the batch size beyond a certain point yields no decrease in wall-clock time to convergence for \emph{either} train or test loss. This batch size is usually substantially below the capacity of current systems. We show that popular training strategies for large batch size optimization begin to fail before we can populate all available compute resources, and we show that the point at which these methods break down depends more on attributes like model architecture and data complexity than it does directly on the size of the dataset.

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• One weird trick for parallelizing convolutional neural networks
• Don't Decay the Learning Rate, Increase the Batch Size
• Large Batch Training of Convolutional Networks
• Pointer Sentinel Mixture Models
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Three Factors Influencing Minima in SGD
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• A Walk with SGD
• How to scale distributed deep learning?
• Large Scale Language Modeling: Converging on 40GB of Text in Four Hours

Cited by in corpus (22)

• Don't Use Large Mini-Batches, Use Local SGD
• Measuring the Effects of Data Parallelism on Neural Network Training
• Large batch size training of neural networks with adversarial training and second-order information
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
• Communication-efficient distributed SGD with Sketching
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• A Study of Gradient Variance in Deep Learning
• Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification
• Distributed Stochastic Algorithms for High-rate Streaming Principal Component Analysis
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Stochastic Training is Not Necessary for Generalization
• Improved generalization by noise enhancement
• Layered gradient accumulation and modular pipeline parallelism: fast and efficient training of large language models
• Inefficiency of K-FAC for Large Batch Size Training
• On Large-Cohort Training for Federated Learning
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Training Efficiency and Robustness in Deep Learning
• Batch size-invariance for policy optimization