55 citations · 138 across the 7 of their papers we have counts for
6 papers · 1 filter
Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
Daniele De Sensi, Lorenzo Pichetti, Flavio Vella +11
Multi-GPU nodes are increasingly common in the rapidly evolving landscape of exascale supercomputers. On these systems, GPUs on the same node are connected through dedicated networ…
Network-Offloaded Bandwidth-Optimal Broadcast and Allgather for Distributed AI
Mikhail Khalilov, Salvatore Di Girolamo, Marcin Chrapek +3
In the Fully Sharded Data Parallel (FSDP) training pipeline, collective operations can be interleaved to maximize the communication/computation overlap. In this scenario, outstandi…
Canary: Congestion-Aware In-Network Allreduce Using Dynamic Trees
Daniele De Sensi, Edgar Costa Molero, Salvatore Di Girolamo +2
The allreduce operation is an essential building block for many distributed applications, ranging from the training of deep learning models to scientific computing. In an allreduce…
Flare: Flexible In-Network Allreduce
Daniele De Sensi, Salvatore Di Girolamo, Saleh Ashkboos +2
The allreduce operation is one of the most commonly used communication routines in distributed applications. To improve its bandwidth and to reduce network traffic, this operation…
An In-Depth Analysis of the Slingshot Interconnect
Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon +2
The interconnect is one of the most critical components in large scale computing systems, and its impact on the performance of applications is going to increase with the system siz…
Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
Shigang Li, Tal Ben-Nun, Salvatore Di Girolamo +2
Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself. Traditional synch…