Sharing a Fabric with Collective Communication: Two Storage Penalties in Deep Learning Training

Chen Wang, Wenzhao Wu, Hyojin Kim +1

Distributed DL training on HPC systems often shares one network fabric between NCCL/RCCL collective communication and parallel-filesystem I/O. Using a real GNN training workload on…