Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect

Abstract

High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and planet-scale simulations. However, the lack of deep understanding on how modern GPUs can be connected and the real impact of state-of-the-art interconnect technology on multi-GPU application performance become a hurdle. In this paper, we fill the gap by conducting a thorough evaluation on five latest types of modern GPU interconnects: PCIe, NVLink-V1, NVLink-V2, NVLink-SLI and NVSwitch, from six high-end servers and HPC platforms: NVIDIA P100-DGX-1, V100-DGX-1, DGX-2, OLCF's SummitDev and Summit supercomputers, as well as an SLI-linked system with two NVIDIA Turing RTX-2080 GPUs. Based on the empirical evaluation, we have observed four new types of GPU communication network NUMA effects: three are triggered by NVLink's topology, connectivity and routing, while one is caused by PCIe chipset design issue. These observations indicate that, for an application running in a multi-GPU node, choosing the right GPU combination can impose considerable impact on GPU communication efficiency, as well as the application's overall performance. Our evaluation can be leveraged in building practical multi-GPU performance models, which are vital for GPU task allocation, scheduling and migration in a shared environment (e.g., AI cloud and HPC centers), as well as communication-oriented performance tuning.

15 pages. The paper is going to be submitted to TPDS

References in corpus (3)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?

Cited by in corpus (27)

• Synthesizing Optimal Collective Algorithms
• Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
• Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric Behaviors
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms
• PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
• Scalable and accurate multi-GPU based image reconstruction of large-scale ptychography data
• Harmony: Overcoming the Hurdles of GPU Memory Capacity to Train Massive DNN Models on Commodity Servers
• Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
• G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations
• iFDK: A Scalable Framework for Instant High-resolution Image Reconstruction
• MAPA: Multi-Accelerator Pattern Allocation Policy for Multi-Tenant GPU Servers
• Large-Scale Discrete Fourier Transform on TPUs
• An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
• Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)
• Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUs
• Improving Energy Saving of One-sided Matrix Decompositions on CPU-GPU Heterogeneous Systems
• Accelerating Multigrid-based Hierarchical Scientific Data Refactoring on GPUs
• Accelerating Recommender Systems via Hardware "scale-in"
• CGX: Adaptive System Support for Communication-Efficient Deep Learning
• ARENA: Asynchronous Reconfigurable Accelerator Ring to Enable Data-Centric Parallel Computing
• Oases: Efficient Large-Scale Model Training on Commodity Servers via Overlapped and Automated Tensor Model Parallelism
• DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging
• PoCL-R: An Open Standard Based Offloading Layer for Heterogeneous Multi-Access Edge Computing with Server Side Scalability
• CaPGNN: Optimizing Parallel Graph Neural Network Training with Joint Caching and Resource-Aware Graph Partitioning
• Scalable Multigrid-based Hierarchical Scientific Data Refactoring on GPUs
• ButterFly BFS -- An Efficient Communication Pattern for Multi Node Traversals