PyTorch Distributed: Experiences on Accelerating Data Parallel Training

Abstract

This paper presents the design, implementation, and evaluation of the PyTorch distributed data parallel module. PyTorch is a widely-adopted scientific computing package used in deep learning research and applications. Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources. Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability. In general, the technique of distributed data parallelism replicates the model on every computational resource to generate gradients independently and then communicates those gradients at each iteration to keep model replicas consistent. Despite the conceptual simplicity of the technique, the subtle dependencies between computation and communication make it non-trivial to optimize the distributed training efficiency. As of v1.5, PyTorch natively provides several techniques to accelerate distributed data parallel, including bucketing gradients, overlapping computation with communication, and skipping gradient synchronization. Evaluations show that, when configured appropriately, the PyTorch distributed data parallel module attains near-linear scalability using 256 GPUs.

To appear in VLDB 2020

References in corpus (6)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• End to End Learning for Self-Driving Cars
• Horovod: fast and easy distributed deep learning in TensorFlow
• Reducing Transformer Depth on Demand with Structured Dropout
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum

Cited by in corpus (27)

• HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• Query Processing on Tensor Computation Runtimes
• FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
• DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs
• On the Utility of Gradient Compression in Distributed Training Systems
• Experimenting with Emerging RISC-V Systems for Decentralised Machine Learning
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers
• e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce
• Taming Momentum in a Distributed Asynchronous Environment
• A Tensor Compiler for Unified Machine Learning Prediction Serving
• ScaDLES: Scalable Deep Learning over Streaming data at the Edge
• Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)
• EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
• Distributed Deep Learning in Open Collaborations
• FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Framework for Heterogeneous Edge Devices
• VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
• Tensor Relational Algebra for Machine Learning System Design
• CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Manner
• Scheduling Optimization Techniques for Neural Network Training
• Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks
• Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems
• Graph Neural Network Training with Data Tiering
• Trade-offs of Local SGD at Scale: An Empirical Study
• High Throughput Training of Deep Surrogates from Large Ensemble Runs
• MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training