Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash

Abstract

Scaling the distributed deep learning to a massive GPU cluster level is challenging due to the instability of the large mini-batch training and the overhead of the gradient synchronization. We address the instability of the large mini-batch training with batch-size control and label smoothing. We address the overhead of the gradient synchronization with 2D-Torus all-reduce. Specifically, 2D-Torus all-reduce arranges GPUs in a logical 2D grid and performs a series of collective operation in different orientations. These two techniques are implemented with Neural Network Libraries (NNL). We have successfully trained ImageNet/ResNet-50 in 122 seconds without significant accuracy loss on ABCI cluster.

References in corpus (10)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Mixed Precision Training
• Don't Decay the Learning Rate, Increase the Batch Size
• Large Batch Training of Convolutional Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Image Classification at Supercomputer Scale
• ImageNet Training in Minutes
• Large batch size training of neural networks with adversarial training and second-order information

Cited by in corpus (29)

• Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect
• MLPerf Training Benchmark
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Optimization for deep learning: theory and algorithms
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds
• Communication optimization strategies for distributed deep neural network training: A survey
• Augment your batch: better training with larger batches
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• Large batch size training of neural networks with adversarial training and second-order information
• Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• On the Utility of Gradient Compression in Distributed Training Systems
• Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training
• Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)
• EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
• Stochastic Training is Not Necessary for Generalization
• LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Models
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware
• Distributed Deep Learning in Open Collaborations
• Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
• Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Large Batch Training Does Not Need Warmup
• A Generalization of the Allreduce Operation
• Secure Distributed Training at Scale
• Oscars: Adaptive Semi-Synchronous Parallel Model for Distributed Deep Learning with Global View
• Scaling the training of particle classification on simulated MicroBooNE events to multiple GPUs