Revisiting Distributed Synchronous SGD

Abstract

Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies.

This article will be superseded by arXiv:1604.00981

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• DRAW: A Recurrent Neural Network For Image Generation
• Exploring the Limits of Language Modeling
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants
• Deep learning with Elastic Averaging SGD
• Splash: User-friendly Programming Interface for Parallelizing Stochastic Algorithms
• ASAGA: Asynchronous Parallel SAGA

Cited by in corpus (196)

• TensorFlow: A system for large-scale machine learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Learning with Non-IID Data
• A Survey on Distributed Machine Learning
• How To Backdoor Federated Learning
• Large Batch Training of Convolutional Networks
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Precision Health Data: Requirements, Challenges and Existing Techniques for Data Security and Privacy
• Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
• Revisiting Small Batch Training for Deep Neural Networks
• Multi-Armed Bandit Based Client Scheduling for Federated Learning
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Split learning for health: Distributed deep learning without sharing raw patient data
• A Field Guide to Federated Optimization
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• Analysis and Optimal Edge Assignment For Hierarchical Federated Learning on Non-IID Data
• Measuring the Effects of Data Parallelism on Neural Network Training
• Large scale distributed neural network training through online distillation
• Database Meets Deep Learning: Challenges and Opportunities
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• BigDL: A Distributed Deep Learning Framework for Big Data
• Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
• Edge Intelligence: Architectures, Challenges, and Applications
• Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Adaptive Federated Learning in Resource Constrained Edge Computing Systems
• Split Learning for collaborative deep learning in healthcare
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Scaling Deep Learning on GPU and Knights Landing clusters
• Benchmarking State-of-the-Art Deep Learning Software Tools
• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
• Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
• A Progressive Batching L-BFGS Method for Machine Learning
• ChainerMN: Scalable Distributed Deep Learning Framework
• Minimizing Latency for Secure Coded Computing Using Secret Sharing via Staircase Codes
• Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
• Training Neural Networks Using Features Replay
• Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs
• Straggler-aware Distributed Learning: Communication Computation Latency Trade-off
• ImageNet Training in Minutes
• Gmail Smart Compose: Real-Time Assisted Writing
• Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning
• ErasureHead: Distributed Gradient Descent without Delays Using Approximate Gradient Coding
• Priority-based Parameter Propagation for Distributed DNN Training
• Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settings
• PPFL: Privacy-preserving Federated Learning with Trusted Execution Environments
• Improving Electron Micrograph Signal-to-Noise with an Atrous Convolutional Encoder-Decoder
• Distributed learning with compressed gradients
• Local Differential Privacy based Federated Learning for Internet of Things
• Semi-Dynamic Load Balancing: Efficient Distributed Learning in Non-Dedicated Environments
• Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version)
• Federated Learning with Buffered Asynchronous Aggregation
• Fairness and Accuracy in Federated Learning
• DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
• Faster On-Device Training Using New Federated Momentum Algorithm
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train
• OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
• Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning
• Stochastic, Distributed and Federated Optimization for Machine Learning
• SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Chainer: A Deep Learning Framework for Accelerating the Research Cycle
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• Training Large Neural Networks with Constant Memory using a New Execution Algorithm
• FEDZIP: A Compression Framework for Communication-Efficient Federated Learning
• Learning from History for Byzantine Robust Optimization
• Deep LSTM for Large Vocabulary Continuous Speech Recognition
• Themis: Fair and Efficient GPU Cluster Scheduling
• Asynchronous Accelerated Proximal Stochastic Gradient for Strongly Convex Distributed Finite Sums
• FedEval: A Holistic Evaluation Framework for Federated Learning
• Gradient Diversity: a Key Ingredient for Scalable Distributed Learning
• Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads
• MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for scaling Deep Learning
• Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models
• Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise
• Distilling Large Language Models into Tiny and Effective Students using pQRNN
• An Accelerated Decentralized Stochastic Proximal Algorithm for Finite Sums
• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of Stragglers
• Accelerating Vertical Federated Learning
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
• Taming Momentum in a Distributed Asynchronous Environment
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• Fully Decoupled Neural Network Learning Using Delayed Gradients
• Straggler-Resilient Distributed Machine Learning with Dynamic Backup Workers
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis
• Block-diagonal Hessian-free Optimization for Training Neural Networks
• Performance Analysis and Comparison of Distributed Machine Learning Systems
• Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos
• Distributed Training Large-Scale Deep Architectures
• Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning
• Parallel Training of Deep Networks with Local Updates
• A Distributed Hierarchical SGD Algorithm with Sparse Global Reduction
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Bandwidth Reduction using Importance Weighted Pruning on Ring AllReduce
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
• A(DP)${}^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy
• Heterogeneity-Aware Asynchronous Decentralized Training
• Advances in Asynchronous Parallel and Distributed Optimization
• Parareal Neural Networks Emulating a Parallel-in-time Algorithm
• At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?
• Constrained Generative Adversarial Network Ensembles for Sharable Synthetic Data Generation
• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
• Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks
• Efficient Privacy Preserving Edge Computing Framework for Image Classification
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
• Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks
• Newton-ADMM: A Distributed GPU-Accelerated Optimizer for Multiclass Classification Problems
• Learning Object-conditioned Exploration using Distributed Soft Actor Critic
• Pufferfish: Communication-efficient Models At No Extra Cost
• Diversity/Parallelism Trade-off in Distributed Systems with Redundancy
• Harnessing the Power of Serverless Runtimes for Large-Scale Optimization
• VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
• Asynchronous Decentralized Learning of a Neural Network
• Speeding up Deep Learning with Transient Servers
• Stanza: Layer Separation for Distributed Training in Deep Learning
• Trends and Advancements in Deep Neural Network Communication
• Dynamic Stale Synchronous Parallel Distributed Training for Deep Learning
• Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers
• ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training
• Collage Inference: Using Coded Redundancy for Low Variance Distributed Image Classification
• MixML: A Unified Analysis of Weakly Consistent Parallel Learning
• KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics
• Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent
• ASAP: Asynchronous Approximate Data-Parallel Computation
• D-SPIDER-SFO: A Decentralized Optimization Algorithm with Faster Convergence Rate for Nonconvex Problems
• Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators
• Device Scheduling and Update Aggregation Policies for Asynchronous Federated Learning
• Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
• Hydra: A Peer to Peer Distributed Training & Data Collection Framework
• Anytime Stochastic Gradient Descent: A Time to Hear from all the Workers
• Heterogeneity-aware Gradient Coding for Straggler Tolerance
• Distributed Deep Learning Strategies For Automatic Speech Recognition
• Federated Generative Adversarial Learning
• Optimizing Redundancy Levels in Master-Worker Compute Clusters for Straggler Mitigation
• DINO: Distributed Newton-Type Optimization Method
• Near-Data Processing for Differentiable Machine Learning Models
• Exponential Moving Average Model in Parallel Speech Recognition Training
• Adaptive Periodic Averaging: A Practical Approach to Reducing Communication in Distributed Learning
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
• Importance Weighted Evolution Strategies
• Accelerating Federated Learning in Heterogeneous Data and Computational Environments
• Heterogeneous CPU+GPU Stochastic Gradient Descent Algorithms
• Joint Parameter-and-Bandwidth Allocation for Improving the Efficiency of Partitioned Edge Learning
• Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes
• Accelerated Sparsified SGD with Error Feedback
• SWIFT: Expedited Failure Recovery for Large-scale DNN Training
• Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning
• Making Asynchronous Stochastic Gradient Descent Work for Transformers
• Improving Efficiency in Large-Scale Decentralized Distributed Training
• Incentive-based integration of useful work into blockchains
• Machine Learning on Volatile Instances
• Straggler-Robust Distributed Optimization in Parameter-Server Networks
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• Gear Training: A new way to implement high-performance model-parallel training
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• Adaptive Braking for Mitigating Gradient Delay
• Differential Privacy Meets Federated Learning under Communication Constraints
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training
• Robust Federated Learning with Noisy Communication
• High Throughput Synchronous Distributed Stochastic Gradient Descent
• A Stochastic Large-scale Machine Learning Algorithm for Distributed Features and Observations
• The Virtual Patch Clamp: Imputing C. elegans Membrane Potentials from Calcium Imaging
• Gap Aware Mitigation of Gradient Staleness
• Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD
• Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping
• Task allocation for decentralized training in heterogeneous environment
• Slim-DP: A Light Communication Data Parallelism for DNN
• Elastic Bulk Synchronous Parallel Model for Distributed Deep Learning
• A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training
• How to Train your DNN: The Network Operator Edition
• Towards Understanding Acceleration Tradeoff between Momentum and Asynchrony in Nonconvex Stochastic Optimization
• Management of Resource at the Network Edge for Federated Learning
• Large-Scale Stochastic Learning using GPUs
• Straggler-Robust Distributed Optimization with the Parameter Server Utilizing Coded Gradient
• Weighted Aggregating Stochastic Gradient Descent for Parallel Deep Learning
• A Highly Efficient Distributed Deep Learning System For Automatic Speech Recognition
• Towards Explainable Multi-Party Learning: A Contrastive Knowledge Sharing Framework
• ${\text{A}}^2{\text{CiD}}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning
• Privacy-Preserving Deep Learning Computation for Geo-Distributed Medical Big-Data Platforms
• Distributed Dual Coordinate Ascent in General Tree Networks and Communication Network Effect on Synchronous Machine Learning