Better Convergence Guarantees for Sign-Based Momentum Methods

Abstract

This paper presents an improved analysis for sign-based methods with momentum updates. Traditional sign-based methods obtain a convergence rate of $\mathcal{O}(T^{-1/4})$ under the separable smoothness assumption, but they typically require large batch sizes or assume unimodal symmetric stochastic noise. To address these limitations, we demonstrate that signSGD with momentum can achieve the same convergence rate using constant batch sizes without additional assumptions. We also establish a convergence rate under the $l_2$-smoothness condition, improving upon the result of prior work by a factor of $\mathcal{O}(d^{1/2})$, where $d$ is the problem dimension. Furthermore, we explore sign-based methods in distributed settings and show that the proposed methods yield convergence rates of $\mathcal{O}\left(d^{1/2}{T}^{-1/2}+d{n}^{-1/2}\right)$ and $\mathcal{O}\left(d^{1/4}{T}^{-1/4}\right)$, which outperform the previous results of $\mathcal{O}\left(d{T}^{-1/4}+d{n}^{-1/2}\right)$ and $\mathcal{O}\left(d^{3/8}{T}^{-1/8}\right)$, respectively. Numerical experiments also validate the effectiveness of the proposed methods.