Distributed Deep Learning Using Synchronous Stochastic Gradient Descent

Abstract

We design and implement a distributed multinode synchronous SGD algorithm, without altering hyper parameters, or compressing data, or altering algorithmic behavior. We perform a detailed analysis of scaling, and identify optimal design points for different networks. We demonstrate scaling of CNNs on 100s of nodes, and present what we believe to be record training throughputs. A 512 minibatch VGG-A CNN training run is scaled 90X on 128 nodes. Also 256 minibatch VGG-A and OverFeat-FAST networks are scaled 53X and 42X respectively on a 64 node cluster. We also demonstrate the generality of our approach via best-in-class 6.5X scaling for a 7-layer DNN on 16 nodes. Thereafter we attempt to democratize deep-learning by training on an Ethernet based AWS cluster and show ~14X scaling on 16 nodes.

References in corpus (2)

• Deep Learning in Neural Networks: An Overview
• Deep Image: Scaling up Image Recognition

Cited by in corpus (45)

• Revisiting Distributed Synchronous SGD
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Revisiting Small Batch Training for Deep Neural Networks
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• An Exhaustive Survey on P4 Programmable Data Plane Switches: Taxonomy, Applications, Challenges, and Future Trends
• Review: Deep Learning in Electron Microscopy
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
• Benchmarking State-of-the-Art Deep Learning Software Tools
• A Progressive Batching L-BFGS Method for Machine Learning
• Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
• ImageNet Training in Minutes
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms
• Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
• On Scale-out Deep Learning Training for Cloud and HPC
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• Sparse Communication for Training Deep Networks
• Exploring the Design Space of Deep Convolutional Neural Networks at Large Scale
• Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory
• Block-diagonal Hessian-free Optimization for Training Neural Networks
• Parallel Training of Deep Networks with Local Updates
• A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks
• Bandwidth Reduction using Importance Weighted Pruning on Ring AllReduce
• SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving
• Compiler-Level Matrix Multiplication Optimization for Deep Learning
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• Array Languages Make Neural Networks Fast
• On Consensus-Optimality Trade-offs in Collaborative Deep Learning
• The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study
• Parallel Complexity of Forward and Backward Propagation
• Asynchronous Decentralized Learning of a Neural Network
• TAG: Gradient Attack on Transformer-based Language Models
• Model Fusion via Neuron Transplantation
• GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent
• Backdrop: Stochastic Backpropagation
• Mini-batch Serialization: CNN Training with Inter-layer Data Reuse
• Distributed stochastic optimization for deep learning (thesis)
• Integrated Model, Batch and Domain Parallelism in Training Neural Networks