Large Batch Training of Convolutional Networks

Abstract

A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• One weird trick for parallelizing convolutional neural networks
• Revisiting Distributed Synchronous SGD
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Diagonal Rescaling For Neural Networks

Cited by in corpus (199)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Don't Decay the Learning Rate, Increase the Batch Size
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Exploring Simple Siamese Representation Learning
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Learned Step Size Quantization
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• High-Performance Large-Scale Image Recognition Without Normalization
• Which picker fits my data? A quantitative evaluation of deep learning based seismic pickers
• Don't Use Large Mini-Batches, Use Local SGD
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Masked Autoencoders Are Scalable Vision Learners
• Tiny Machine Learning: Progress and Futures
• MLPerf Training Benchmark
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Self-supervised Pretraining of Visual Features in the Wild
• An Empirical Model of Large-Batch Training
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Supervised Contrastive Learning
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Adversarial Self-Supervised Contrastive Learning
• Spatiotemporal Contrastive Video Representation Learning
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Communication optimization strategies for distributed deep neural network training: A survey
• Implicit Regularization in Deep Matrix Factorization
• Intriguing Properties of Contrastive Losses
• DualNet: Continual Learning, Fast and Slow
• A Progressive Batching L-BFGS Method for Machine Learning
• Deep Learning for EEG Seizure Detection in Preterm Infants
• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• Contrastive Training for Improved Out-of-Distribution Detection
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• A Unified and Refined Convergence Analysis for Non-Convex Decentralized Learning
• Augment your batch: better training with larger batches
• An Exponential Learning Rate Schedule for Deep Learning
• Whitening for Self-Supervised Representation Learning
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Representation Learning via Invariant Causal Mechanisms
• Federated Learning with Buffered Asynchronous Aggregation
• PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentation
• Faster On-Device Training Using New Federated Momentum Algorithm
• The Implicit Regularization of Stochastic Gradient Flow for Least Squares
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Large batch size training of neural networks with adversarial training and second-order information
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
• Tesseract: Parallelize the Tensor Parallelism Efficiently
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning
• Understanding Generalization through Visualizations
• CosmoFlow: Using Deep Learning to Learn the Universe at Scale
• BYOL works even without batch statistics
• Maximizing Parallelism in Distributed Training for Huge Neural Networks
• LoCo: Local Contrastive Representation Learning
• On the Utility of Gradient Compression in Distributed Training Systems
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Does label smoothing mitigate label noise?
• BenthicNet: A global compilation of seafloor images for deep learning applications
• Robust Distributed Optimization With Randomly Corrupted Gradients
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• Communication-efficient distributed SGD with Sketching
• Learning From Long-Tailed Data With Noisy Labels
• Anatomically aware dual-hop learning for pulmonary embolism detection in CT pulmonary angiograms
• DECAR: Deep Clustering for learning general-purpose Audio Representations
• Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
• CLAR: Contrastive Learning of Auditory Representations
• Pre-Trained Models: Past, Present and Future
• Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• The Convergence of Sparsified Gradient Methods
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Adam${}^{+}$: A Stochastic Method with Adaptive Variance Reduction
• UFO-ViT: High Performance Linear Vision Transformer without Softmax
• SparCML: High-Performance Sparse Communication for Machine Learning
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• Multiplicative update rules for accelerating deep learning training and increasing robustness
• Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning
• Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification
• Temperature as Uncertainty in Contrastive Learning
• Contrastive Learning Improves Model Robustness Under Label Noise
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
• Exploring the limits of Concurrency in ML Training on Google TPUs
• Improved Analysis of Clipping Algorithms for Non-convex Optimization
• Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
• RegionCL: Can Simple Region Swapping Contribute to Contrastive Learning?
• Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search
• Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos
• Revisiting One-vs-All Classifiers for Predictive Uncertainty and Out-of-Distribution Detection in Neural Networks
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Non-contrastive representation learning for intervals from well logs
• Towards Domain-Agnostic Contrastive Learning
• Improving Transformation Invariance in Contrastive Representation Learning
• InfoCNF: An Efficient Conditional Continuous Normalizing Flow with Adaptive Solvers
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
• Automated Learning Rate Scheduler for Large-batch Training
• MixSiam: A Mixture-based Approach to Self-supervised Representation Learning
• Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training
• Disentangling Adaptive Gradient Methods from Learning Rates
• Contrastive Representation Learning for Whole Brain Cytoarchitectonic Mapping in Histological Human Brain Sections
• BroadFace: Looking at Tens of Thousands of People at Once for Face Recognition
• Identifying Learning Rules From Neural Network Observables
• Study on the Large Batch Size Training of Neural Networks Based on the Second Order Gradient
• PANDA: Facilitating Usable AI Development
• Neural networks for semantic segmentation of historical city maps: Cross-cultural performance and the impact of figurative diversity
• Representation Learning via Consistent Assignment of Views to Clusters
• Stochastic Training is Not Necessary for Generalization
• Self-Supervised Learning by Estimating Twin Class Distributions
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• HierTrain: Fast Hierarchical Edge AI Learning with Hybrid Parallelism in Mobile-Edge-Cloud Computing
• HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow
• Parallel Complexity of Forward and Backward Propagation
• Embedding Transfer with Label Relaxation for Improved Metric Learning
• S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
• VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
• ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training
• Self-supervised Representation Learning with Relative Predictive Coding
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• On Feature Decorrelation in Self-Supervised Learning
• Stochastic Optimization with Laggard Data Pipelines
• Layerwise Optimization by Gradient Decomposition for Continual Learning
• Distributed Deep Learning in Open Collaborations
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• A Generalizable Approach to Learning Optimizers
• CRNNs for Urban Sound Tagging with spatiotemporal context
• COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Fast, Better Training Trick -- Random Gradient
• Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD
• High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails
• MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems
• Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
• On Large-Cohort Training for Federated Learning
• Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks
• Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability
• A Unified Mixture-View Framework for Unsupervised Representation Learning
• A Fast Knowledge Distillation Framework for Visual Recognition
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Train Feedfoward Neural Network with Layer-wise Adaptive Rate via Approximating Back-matching Propagation
• Heterogeneous Contrastive Learning
• Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation
• Concurrent Adversarial Learning for Large-Batch Training
• Multi-Plateau Ensemble for Endoscopic Artefact Segmentation and Detection
• Compressive Visual Representations
• Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection
• Extrapolation for Large-batch Training in Deep Learning
• Large Batch Training Does Not Need Warmup
• DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging
• Heavy-tailed Streaming Statistical Estimation
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Multirate Training of Neural Networks
• Supervised Momentum Contrastive Learning for Few-Shot Classification
• CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
• NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning
• Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters
• UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
• Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping
• OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training
• Scaling Distributed Training with Adaptive Summation
• Gradient Sparification for Asynchronous Distributed Training
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• High Throughput Synchronous Distributed Stochastic Gradient Descent
• Blockwise Adaptivity: Faster Training and Better Generalization in Deep Learning
• Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clipping
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• Trade-offs of Local SGD at Scale: An Empirical Study
• Logit Attenuating Weight Normalization
• What can linear interpolation of neural network loss landscapes tell us?
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
• MIO : Mutual Information Optimization using Self-Supervised Binary Contrastive Learning
• Slim-DP: A Light Communication Data Parallelism for DNN
• Supervised Tree-Wasserstein Distance
• ScaleNet: An Unsupervised Representation Learning Method for Limited Information
• Representation Synthesis by Probabilistic Many-Valued Logic Operation in Self-Supervised Learning
• ${\text{A}}^2{\text{CiD}}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning
• Bridging the Gap between Deep Learning and Frustrated Quantum Spin System for Extreme-scale Simulations on New Generation of Sunway Supercomputer
• Offline Clustering Approach to Self-supervised Learning for Class-imbalanced Image Data
• Addressing Algorithmic Bottlenecks in Elastic Machine Learning with Chicle
• Data optimization for large batch distributed training of deep neural networks
• Learning Metrics from Mean Teacher: A Supervised Learning Method for Improving the Generalization of Speaker Verification System
• Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations
• Convergence Analysis of Gradient Descent Algorithms with Proportional Updates
• Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory