Rethinking Machine Learning Collective Communication as a Multi-Commodity Flow Problem

Behnaz Arzani, Siva Kesava Reddy Kakarla, Miguel Castro +3

We show communication schedulers' recent work proposed for ML collectives does not scale to the increasing problem sizes that arise from training larger models. These works also of…