Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

Abstract

We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.

NIPS'17 Workshop: Deep Learning at Supercomputer Scale

References in corpus (3)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ChainerMN: Scalable Distributed Deep Learning Framework
• ImageNet Training in Minutes

Cited by in corpus (34)

• Improving Generalization Performance by Switching from Adam to SGD
• An Empirical Model of Large-Batch Training
• Optimization for deep learning: theory and algorithms
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• ChainerMN: Scalable Distributed Deep Learning Framework
• ImageNet Training in Minutes
• TF-Replicator: Distributed Machine Learning for Researchers
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• Large-Batch Training for LSTM and Beyond
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
• Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training
• Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
• A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization
• HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking
• Speeding up Deep Learning with Transient Servers
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• Fast Training of Sparse Graph Neural Networks on Dense Hardware
• Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning
• Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD
• Wavelet Denoised-ResNet CNN and LightGBM Method to Predict Forex Rate of Change
• Large Batch Training Does Not Need Warmup
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Deep Residual Network based food recognition for enhanced Augmented Reality application
• Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning
• Contrastive Weight Regularization for Large Minibatch SGD
• DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging
• E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
• OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training
• NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse Networks
• AccUDNN: A GPU Memory Efficient Accelerator for Training Ultra-deep Neural Networks
• Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training
• Identification of complex mixtures for Raman spectroscopy using a novel scheme based on a new multi-label deep neural network
• Caramel: Accelerating Decentralized Distributed Deep Learning with Computation Scheduling
• Network Implosion: Effective Model Compression for ResNets via Static Layer Pruning and Retraining