Measuring the Effects of Data Parallelism on Neural Network Training

Abstract

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Exploring the Limits of Language Modeling
• Revisiting Distributed Synchronous SGD
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Deep Learning Scaling is Predictable, Empirically
• Revisiting Small Batch Training for Deep Neural Networks
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Don't Use Large Mini-Batches, Use Local SGD
• Sharp Minima Can Generalize For Deep Nets
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Parallelizing Stochastic Gradient Descent for Least Squares Regression: mini-batching, averaging, and model misspecification
• The Effects of Hyperparameters on SGD Training of Neural Networks
• ImageNet Training in Minutes
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Benchmarking of LSTM Networks
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Critical Hyper-Parameters: No Random, No Cry
• The Effect of Network Width on the Performance of Large-batch Training

Cited by in corpus (89)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Scaling Laws for Neural Language Models
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• Don't Use Large Mini-Batches, Use Local SGD
• Single chip photonic deep neural network with accelerated training
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• An Empirical Model of Large-Batch Training
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Reducing Noise in GAN Training with Variance Reduced Extragradient
• Energy Efficient Federated Learning Over Wireless Communication Networks
• Augment your batch: better training with larger batches
• Four Things Everyone Should Know to Improve Batch Normalization
• On the Origin of Implicit Regularization in Stochastic Gradient Descent
• Federated Learning with Buffered Asynchronous Aggregation
• Large batch size training of neural networks with adversarial training and second-order information
• Faster Neural Network Training with Data Echoing
• Small-GAN: Speeding Up GAN Training Using Core-sets
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
• Training independent subnetworks for robust prediction
• On the different regimes of Stochastic Gradient Descent
• The Scientific Method in the Science of Machine Learning
• Market Driven Multi-domain Network Service Orchestration in 5G Networks
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• BlinkML: Efficient Maximum Likelihood Estimation with Probabilistic Guarantees
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
• Learning from History for Byzantine Robust Optimization
• Consensus Control for Decentralized Deep Learning
• Encoder-Agnostic Adaptation for Conditional Language Generation
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• Exploring Benefits of Transfer Learning in Neural Machine Translation
• Communication-efficient distributed SGD with Sketching
• Passage Ranking with Weak Supervision
• How to decay your learning rate
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• Skyline: Interactive In-Editor Computational Performance Profiling for Deep Neural Network Training
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• Directional convergence and alignment in deep learning
• FuncPipe: A Pipelined Serverless Framework for Fast and Cost-efficient Training of Deep Learning Models
• A Study of Gradient Variance in Deep Learning
• Reducing the variance in online optimization by transporting past gradients
• Sequential Interpretability: Methods, Applications, and Future Direction for Understanding Deep Learning Models in the Context of Sequential Data
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• Large-Batch Training for LSTM and Beyond
• The Two Regimes of Deep Network Training
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Exploring the limits of Concurrency in ML Training on Google TPUs
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Parallel Training of Deep Networks with Local Updates
• Federated Learning on Non-IID Data: A Survey
• Identifying Exoplanets with Deep Learning. IV. Removing Stellar Activity Signals from Radial Velocity Measurements Using Neural Networks
• At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
• Label Noise SGD Provably Prefers Flat Global Minimizers
• Multi-node Bert-pretraining: Cost-efficient Approach
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
• Anti-Distillation: Improving reproducibility of deep networks
• A Loss Curvature Perspective on Training Instability in Deep Learning
• Characterizing Deep Learning Training Workloads on Alibaba-PAI
• VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
• Taming GANs with Lookahead-Minmax
• Stochastic Optimization with Laggard Data Pipelines
• COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression
• Fast Training of Sparse Graph Neural Networks on Dense Hardware
• Multi-Task Prediction of Clinical Outcomes in the Intensive Care Unit using Flexible Multimodal Transformers
• Layered gradient accumulation and modular pipeline parallelism: fast and efficient training of large language models
• Inefficiency of K-FAC for Large Batch Size Training
• Distilling Information from a Flood: A Possibility for the Use of Meta-Analysis and Systematic Review in Machine Learning Research
• Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
• Concurrent Adversarial Learning for Large-Batch Training
• Large Batch Training Does Not Need Warmup
• Extrapolation for Large-batch Training in Deep Learning
• Gap Aware Mitigation of Gradient Staleness
• BPPSA: Scaling Back-propagation by Parallel Scan Algorithm
• Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping
• Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation
• Training Efficiency and Robustness in Deep Learning
• Logit Attenuating Weight Normalization
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
• Trade-offs of Local SGD at Scale: An Empirical Study
• Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
• Batch size-invariance for policy optimization