3 papers
cs.DC2024
Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
Daniele De Sensi, Lorenzo Pichetti, Flavio Vella +11
Multi-GPU nodes are increasingly common in the rapidly evolving landscape of exascale supercomputers. On these systems, GPUs on the same node are connected through dedicated networ…
cs.DC2024
Network-Offloaded Bandwidth-Optimal Broadcast and Allgather for Distributed AI
Mikhail Khalilov, Salvatore Di Girolamo, Marcin Chrapek +3
In the Fully Sharded Data Parallel (FSDP) training pipeline, collective operations can be interleaved to maximize the communication/computation overlap. In this scenario, outstandi…
cs.DC2023
Canary: Congestion-Aware In-Network Allreduce Using Dynamic Trees
Daniele De Sensi, Edgar Costa Molero, Salvatore Di Girolamo +2
The allreduce operation is an essential building block for many distributed applications, ranging from the training of deep learning models to scientific computing. In an allreduce…