Design Space Exploration of DMA based Finer-Grain Compute Communication Overlap

Shagnik Pal, Shaizeen Aga, Suchita Pati +2

Modern ML workloads demand distributing training and inference across multiple GPUs. However, these parallelization techniques often suffer from exposed critical-path communication…