Ordered Local Momentum for Asynchronous Distributed Learning under Arbitrary Delays

Chang-Wei Shi, Shi-Shang Wang, Wu-Jun Li

Momentum SGD (MSGD) serves as a foundational optimizer in training deep models due to momentum's key role in accelerating convergence and enhancing generalization. Meanwhile, async…