Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms

Abstract

Deep Learning (DL) training platforms are built by interconnecting multiple DL accelerators (e.g., GPU/TPU) via fast, customized interconnects with 100s of gigabytes (GBs) of bandwidth. However, as we identify in this work, driving this bandwidth is quite challenging. This is because there is a pernicious balance between using the accelerator's compute and memory for both DL computations and communication. This work makes two key contributions. First, via real system measurements and detailed modeling, we provide an understanding of compute and memory bandwidth demands for DL compute and comms. Second, we propose a novel DL collective communication accelerator called Accelerator Collectives Engine (ACE) that sits alongside the compute and networking engines at the accelerator endpoint. ACE frees up the endpoint's compute and memory resources for DL compute, which in turn reduces the required memory BW by 3.5X on average to drive the same network BW compared to state-of-the-art baselines. For modern DL workloads and different network sizes, ACE, on average, increases the effective network bandwidth utilization by 1.44X (up to 2.67X), resulting in an average of 1.41X (up to 1.51X), 1.12X (up to 1.17X), and 1.13X (up to 1.19X) speedup in iteration time for ResNet-50, GNMT and DLRM when compared to the best baseline configuration, respectively.

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• Scaling Distributed Machine Learning with In-Network Aggregation

Cited by in corpus (8)

• ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale
• Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
• Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models
• DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models
• Improving Automatic Parallel Training via Balanced Memory Workload Optimization
• LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Models
• Towards a Standardized Representation for Deep Learning Collective Algorithms
• TAPAS: Fast and Automatic Derivation of Tensor Parallel Strategies for Large Neural Networks