Controlled Periodic Synchronization for Efficient Data-Parallel Training

Imane Ettifouri, Mostapha Zbakh, Claude Tadonki

Data-parallel training relies on frequent gradient synchronization across workers. Standard DDP synchronizes gradients at every iteration, which is effective on fast local-area net…