Coordination-Free Workload Regularization for Variable-Cost Distributed Training

Tao Huang, Chen Zhu, Antonio Alonso +4

Synchronous data-parallel training usually assigns every rank the same number of records, even when record costs vary by orders of magnitude. For variable-length models, this conve…