ImageNet Training in Minutes

Abstract

Finishing 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU takes 14 days. This training requires 10^18 single precision operations in total. On the other hand, the world's current fastest supercomputer can finish 2 * 10^17 single precision operations per second (Dongarra et al 2017, https://www.top500.org/lists/2017/06/). If we can make full use of the supercomputer for DNN training, we should be able to finish the 90-epoch ResNet-50 training in one minute. However, the current bottleneck for fast DNN training is in the algorithm level. Specifically, the current batch size (e.g. 512) is too small to make efficient use of many processors. For large-scale DNN training, we focus on using large-batch data-parallelism synchronous SGD without losing accuracy in the fixed epochs. The LARS algorithm (You, Gitman, Ginsburg, 2017, arXiv:1708.03888) enables us to scale the batch size to extremely large case (e.g. 32K). We finish the 100-epoch ImageNet training with AlexNet in 11 minutes on 1024 CPUs. About three times faster than Facebook's result (Goyal et al 2017, arXiv:1706.02677), we finish the 90-epoch ImageNet training with ResNet-50 in 20 minutes on 2048 KNLs without losing accuracy. State-of-the-art ImageNet training speed with ResNet-50 is 74.9% top-1 test accuracy in 15 minutes. We got 74.9% top-1 test accuracy in 64 epochs, which only needs 14 minutes. Furthermore, when we increase the batch size to above 16K, our accuracy is much higher than Facebook's on corresponding batch sizes. Our source code is available upon request.

References in corpus (10)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• One weird trick for parallelizing convolutional neural networks
• Revisiting Distributed Synchronous SGD
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Deep learning with Elastic Averaging SGD

Cited by in corpus (27)

• Don't Decay the Learning Rate, Increase the Batch Size
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Measuring the Effects of Data Parallelism on Neural Network Training
• BigDL: A Distributed Deep Learning Framework for Big Data
• An Empirical Model of Large-Batch Training
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• ShiftAddNet: A Hardware-Inspired Deep Network
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• MegDet: A Large Mini-Batch Object Detector
• Chainer: A Deep Learning Framework for Accelerating the Research Cycle
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
• A(DP)${}^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy
• Fast Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron
• Dynamic Sparse Graph for Efficient Deep Learning
• Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks
• Detecting Forged Facial Videos using convolutional neural network
• High Throughput Synchronous Distributed Stochastic Gradient Descent
• Contrastive Weight Regularization for Large Minibatch SGD
• CNN with large memory layers