Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

Abstract

We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.

arXiv commentNIPS'17 Workshop: Deep Learning at Supercomputer Scale

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• ChainerMN: Scalable Distributed Deep Learning Framework
• ImageNet Training in Minutes

Cited by in corpus (62)

• Don't Decay the Learning Rate, Increase the Batch Size
• Improving Generalization Performance by Switching from Adam to SGD
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Measuring the Effects of Data Parallelism on Neural Network Training
• An Empirical Model of Large-Batch Training
• Optimization for deep learning: theory and algorithms
• Combining a Convolutional Neural Network with Autoencoders to Predict the Survival Chance of COVID-19 Patients
• Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
• Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Communication optimization strategies for distributed deep neural network training: A survey
• ChainerMN: Scalable Distributed Deep Learning Framework
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• ImageNet Training in Minutes
• Tesseract: Parallelize the Tensor Parallelism Efficiently
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• TF-Replicator: Distributed Machine Learning for Researchers
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• FanStore: Enabling Efficient and Scalable I/O for Distributed Deep Learning
• PFDet: 2nd Place Solution to Open Images Challenge 2018 Object Detection Track
• Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• Large-Batch Training for LSTM and Beyond
• SparCML: High-Performance Sparse Communication for Machine Learning
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
• Towards recognizing the light facet of the Higgs Boson
• Automating Vehicles by Deep Reinforcement Learning using Task Separation with Hill Climbing
• Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
• Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training
• A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization
• Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks
• PAFNet: An Efficient Anchor-Free Object Detector Guidance
• Democratizing Production-Scale Distributed Deep Learning
• Speeding up Deep Learning with Transient Servers
• HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking
• Wavelet Denoised-ResNet CNN and LightGBM Method to Predict Forex Rate of Change
• Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
• Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD
• Fast Training of Sparse Graph Neural Networks on Dense Hardware
• Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• Concurrent Adversarial Learning for Large-Batch Training
• Deep Residual Network based food recognition for enhanced Augmented Reality application
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Large Batch Training Does Not Need Warmup
• Accelerated CNN Training Through Gradient Approximation
• NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse Networks
• Contrastive Weight Regularization for Large Minibatch SGD
• AccUDNN: A GPU Memory Efficient Accelerator for Training Ultra-deep Neural Networks
• OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training
• DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
• Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning
• Identification of complex mixtures for Raman spectroscopy using a novel scheme based on a new multi-label deep neural network
• Caramel: Accelerating Decentralized Distributed Deep Learning with Computation Scheduling
• Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training
• Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining
• A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training