Collective Communication Profiling of Modern-day Machine Learning Workloads

Jit Gupta, Andrew Li, Tarun Banka +3

Machine Learning jobs, carried out on large number of distributed high performance systems, involve periodic communication using operations like AllReduce, AllGather, and Broadcast…