Congestion-Aware Path Selection for Load Balancing in AI Clusters

Erfan Nosrati, Majid Ghaderi

Fast training of large machine learning models requires distributed training on AI clusters consisting of thousands of GPUs. The efficiency of distributed training crucially depend…