DeFT: Mitigating Data Dependencies for Flexible Communication Scheduling in Distributed Training

Lin Meng, Yuzhong Sun

Communication scheduling aims to reduce communication bottlenecks in data parallel training (DP) by maximizing the overlap between computation and communication. However, existing…