Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Abstract

Synchronized stochastic gradient descent (SGD) optimizers with data parallelism are widely used in training large-scale deep neural networks. Although using larger mini-batch sizes can improve the system scalability by reducing the communication-to-computation ratio, it may hurt the generalization ability of the models. To this end, we build a highly scalable deep learning training system for dense GPU clusters with three main contributions: (1) We propose a mixed-precision training method that significantly improves the training throughput of a single GPU without losing accuracy. (2) We propose an optimization approach for extremely large mini-batch size (up to 64k) that can train CNN models on the ImageNet dataset without losing accuracy. (3) We propose highly optimized all-reduce algorithms that achieve up to 3x and 11x speedup on AlexNet and ResNet-50 respectively than NCCL-based training on a cluster with 1024 Tesla P40 GPUs. On training ResNet-50 with 90 epochs, the state-of-the-art GPU-based system with 1024 Tesla P100 GPUs spent 15 minutes and achieved 74.9\% top-1 test accuracy, and another KNL-based system with 2048 Intel KNLs spent 20 minutes and achieved 75.4\% accuracy. Our training system can achieve 75.8\% top-1 test accuracy in only 6.6 minutes using 2048 Tesla P40 GPUs. When training AlexNet with 95 epochs, our system can achieve 58.7\% top-1 test accuracy within 4 minutes, which also outperforms all other existing systems.

arXiv admin note: text overlap with arXiv:1803.03383 by other authors

References in corpus (12)

• Mixed Precision Training
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Horovod: fast and easy distributed deep learning in TensorFlow
• Large Batch Training of Convolutional Networks
• Training deep neural networks with low precision multiplications
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• High-Accuracy Low-Precision Training
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning

Cited by in corpus (95)

• Lookahead Optimizer: k steps forward, 1 step back
• MLPerf Training Benchmark
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Database Meets Deep Learning: Challenges and Opportunities
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• Optimization for deep learning: theory and algorithms
• Flower: A Friendly Federated Learning Research Framework
• Image Classification at Supercomputer Scale
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Measuring the Algorithmic Efficiency of Neural Networks
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Learning 2-opt Heuristics for the Traveling Salesman Problem via Deep Reinforcement Learning
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Understanding Top-k Sparsification in Distributed Deep Learning
• Communication optimization strategies for distributed deep neural network training: A survey
• Convergence of Artificial Intelligence and High Performance Computing on NSF-supported Cyberinfrastructure
• Augment your batch: better training with larger batches
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Semi-Dynamic Load Balancing: Efficient Distributed Learning in Non-Dedicated Environments
• Large batch size training of neural networks with adversarial training and second-order information
• Tesseract: Parallelize the Tensor Parallelism Efficiently
• DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
• SPRING: A Sparsity-Aware Reduced-Precision Monolithic 3D CNN Accelerator Architecture for Training and Inference
• Deep Leakage from Gradients
• TF-Replicator: Distributed Machine Learning for Researchers
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• Highly-scalable, physics-informed GANs for learning solutions of stochastic PDEs
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
• Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
• NetReduce: RDMA-Compatible In-Network Reduction for Distributed DNN Training Acceleration
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training
• An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
• Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
• Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks
• Accelerating CNN Training by Pruning Activation Gradients
• Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training
• Heterogeneity-Aware Asynchronous Decentralized Training
• Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
• Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks
• Stochastic Training is Not Necessary for Generalization
• EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
• Democratizing Production-Scale Distributed Deep Learning
• The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study
• HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow
• MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms
• VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
• HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking
• Fast Training of Sparse Graph Neural Networks on Dense Hardware
• Distributed Deep Learning Strategies For Automatic Speech Recognition
• Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
• Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning
• Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
• Inefficiency of K-FAC for Large Batch Size Training
• Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
• Hydra: A Peer to Peer Distributed Training & Data Collection Framework
• Communication-efficient Decentralized Machine Learning over Heterogeneous Networks
• Accelerated CNN Training Through Gradient Approximation
• A Simple Non-i.i.d. Sampling Approach for Efficient Training and Better Generalization
• Selective sampling for accelerating training of deep neural networks
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Large Batch Training Does Not Need Warmup
• Concurrent Adversarial Learning for Large-Batch Training
• Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability
• Improving compute efficacy frontiers with SliceOut
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration
• Complexity-aware Adaptive Training and Inference for Edge-Cloud Distributed AI Systems
• Asynchronous Decentralized Distributed Training of Acoustic Models
• GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent
• On-device Federated Learning with Flower
• A Generalization of the Allreduce Operation
• DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging
• Shuffle-Exchange Brings Faster: Reduce the Idle Time During Communication for Decentralized Neural Network Training
• E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
• Gradient Sparification for Asynchronous Distributed Training
• Distributed Training of Deep Neural Network Acoustic Models for Automatic Speech Recognition
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline
• OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training
• Representation range needs for 16-bit neural network training
• HPSGD: Hierarchical Parallel SGD With Stale Gradients Featuring
• MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning
• Data optimization for large batch distributed training of deep neural networks
• Oscars: Adaptive Semi-Synchronous Parallel Model for Distributed Deep Learning with Global View
• Progressive Compressed Records: Taking a Byte out of Deep Learning Data
• Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks