Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization

Kristi Topollai, Allan Ma, Tolga Dimlioglu +2

Communication-efficient distributed optimizers such as DiLoCo reduce synchronization costs by letting workers perform many local updates before aggregating their progress with an o…