Revisiting Small Batch Training for Deep Neural Networks

Abstract

Modern deep neural network training is typically based on mini-batch stochastic gradient optimization. While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footprint, which might also be exploited to improve machine throughput. In this paper, we review common assumptions on learning rate scaling and training duration, as a basis for an experimental comparison of test performance for different mini-batch sizes. We adopt a learning rate that corresponds to a constant average weight update per gradient calculation (i.e., per unit cost of computation), and point out that this results in a variance of the weight updates that increases linearly with the mini-batch size $m$. The collected experimental results for the CIFAR-10, CIFAR-100 and ImageNet datasets show that increasing the mini-batch size progressively reduces the range of learning rates that provide stable convergence and acceptable test performance. On the other hand, small mini-batch sizes provide more up-to-date gradient calculations, which yields more stable and reliable training. The best performance has been consistently obtained for mini-batch sizes between $m=2$ and $m=32$, which contrasts with recent work advocating the use of mini-batch sizes in the thousands.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• One weird trick for parallelizing convolutional neural networks
• Revisiting Distributed Synchronous SGD
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Three Factors Influencing Minima in SGD
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

Cited by in corpus (93)

• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Don't Use Large Mini-Batches, Use Local SGD
• Measuring the Effects of Data Parallelism on Neural Network Training
• Optimize TSK Fuzzy Systems for Classification Problems: Mini-Batch Gradient Descent with Uniform Regularization and Batch Normalization
• Scaling Distributed Machine Learning with In-Network Aggregation
• Machine Learning the Physical Non-Local Exchange-Correlation Functional of Density-Functional Theory
• Review: Deep Learning in Electron Microscopy
• Fully Automated Multi-Organ Segmentation in Abdominal Magnetic Resonance Imaging with Deep Neural Networks
• Understanding the Role of Training Regimes in Continual Learning
• Change Point Detection in Time Series Data using Autoencoders with a Time-Invariant Representation
• Orthographic Feature Transform for Monocular 3D Object Detection
• A newcomer's guide to deep learning for inverse design in nano-photonics
• Training Neural Response Selection for Task-Oriented Dialogue Systems
• Augment your batch: better training with larger batches
• Relevance of Rotationally Equivariant Convolutions for Predicting Molecular Properties
• Point-Cloud Deep Learning of Porous Media for Permeability Prediction
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Understanding Batch Normalization
• FairBatch: Batch Selection for Model Fairness
• Portraying Double Higgs at the Large Hadron Collider
• Low-Memory Neural Network Training: A Technical Report
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• CASI: A Convolutional Neural Network Approach for Shell Identification
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• Optimized Generic Feature Learning for Few-shot Classification across Domains
• Differentially-Private "Draw and Discard" Machine Learning
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
• Hierarchically modelling Kepler dwarfs and subgiants to improve inference of stellar properties with asteroseismology
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• Beyond Graph Neural Networks with Lifted Relational Neural Networks
• Hot-Starting the Ac Power Flow with Convolutional Neural Networks
• A Study of Gradient Variance in Deep Learning
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• Efficient Learning of Interpretable Classification Rules
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• Domain-specific Communication Optimization for Distributed DNN Training
• Neural Entropic Estimation: A faster path to mutual information estimation
• HyPar: Towards Hybrid Parallelism for Deep Learning Accelerator Array
• Full deep neural network training on a pruned weight budget
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
• Parallel Training of Deep Networks with Local Updates
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Empirical study towards understanding line search approximations for training neural networks
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• Manipulating SGD with Data Ordering Attacks
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Graphcore C2 Card performance for image-based deep learning application: A Report
• Stochastic Training is Not Necessary for Generalization
• Momentum-based variance-reduced proximal stochastic gradient method for composite nonconvex stochastic optimization
• Data-Driven Modeling of Geometry-Adaptive Steady Heat Transfer based on Convolutional Neural Networks: Heat Conduction
• Automated Classification of Helium Ingress in Irradiated X-750
• Automated Copper Alloy Grain Size Evaluation Using a Deep-learning CNN
• Deep learning for pedestrians: backpropagation in CNNs
• Consistency Regularization for Deep Face Anti-Spoofing
• Semi-supervised lung nodule retrieval
• Self-Supervised Learning for Fine-Grained Visual Categorization
• Driver Identification via the Steering Wheel
• Deep Multi-View Learning via Task-Optimal CCA
• On Large-Cohort Training for Federated Learning
• Parallel and distributed asynchronous adaptive stochastic gradient methods
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
• Stochastic Weight Matrix-based Regularization Methods for Deep Neural Networks
• Minibatch Processing in Spiking Neural Networks
• Reliability and Performance Assessment of Federated Learning on Clinical Benchmark Data
• Extrapolation for Large-batch Training in Deep Learning
• Faster SGD training by minibatch persistency
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• Contrastive Weight Regularization for Large Minibatch SGD
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
• NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse Networks
• Distinguishing artefacts: evaluating the saturation point of convolutional neural networks
• Parallelizable Stack Long Short-Term Memory
• GOALS: Gradient-Only Approximations for Line Searches Towards Robust and Consistent Training of Deep Neural Networks
• HyperNP: Interactive Visual Exploration of Multidimensional Projection Hyperparameters
• Unique Properties of Flat Minima in Deep Networks
• Binary Classification as a Phase Separation Process
• Attempted Blind Constrained Descent Experiments
• Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training
• Hybrid Attention Networks for Flow and Pressure Forecasting in Water Distribution Systems
• Modulating Regularization Frequency for Efficient Compression-Aware Model Training
• Training Efficiency and Robustness in Deep Learning
• Convergence of GANs Training: A Game and Stochastic Control Methodology
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training
• Deep Learning in Target Space
• Active Learning for Argument Mining: A Practical Approach
• One-element Batch Training by Moving Window
• Adaptive norms for deep learning with regularized Newton methods
• New Interpretations of Normalization Methods in Deep Learning
• DeepFoldit -- A Deep Reinforcement Learning Neural Network Folding Proteins