ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics

Liangyu Zhao, Saeed Maleki, Yuanhong Wang +4

As modern DNN models grow ever larger, collective communications between the accelerators (allreduce, etc.) emerge as a significant performance bottleneck. Designing efficient comm…