A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network

Abstract

Incorporating a so-called "momentum" dynamic in gradient descent methods is widely used in neural net training as it has been broadly observed that, at least empirically, it often leads to significantly faster convergence. At the same time, there are very few theoretical guarantees in the literature to explain this apparent acceleration effect. Even for the classical strongly convex quadratic problems, several existing results only show Polyak's momentum has an accelerated linear rate asymptotically. In this paper, we first revisit the quadratic problems and show a non-asymptotic accelerated linear rate of Polyak's momentum. Then, we provably show that Polyak's momentum achieves acceleration for training a one-layer wide ReLU network and a deep linear network, which are perhaps the two most popular canonical models for studying optimization and deep learning in the literature. Prior work Du at al. 2019 and Wu et al. 2019 showed that using vanilla gradient descent, and with an use of over-parameterization, the error decays as $(1-\Theta (\frac{1}{{\kappa }^{\prime }}){)}^t$ after $t$ iterations, where ${\kappa }^{\prime }$ is the condition number of a Gram Matrix. Our result shows that with the appropriate choice of parameters Polyak's momentum has a rate of $(1-\Theta (\frac{1}{\sqrt{{\kappa }^{\prime }}}){)}^t$. For the deep linear network, prior work Hu et al. 2020 showed that vanilla gradient descent has a rate of $(1-\Theta (\frac{1}{\kappa }){)}^t$, where $\kappa$ is the condition number of a data matrix. Our result shows an acceleration rate $(1-\Theta (\frac{1}{\sqrt{\kappa }}){)}^t$ is achievable by Polyak's momentum. All the results in this work are obtained from a modular analysis, which can be of independent interest. This work establishes that momentum does indeed speed up neural net training.

Accepted at ICML 2021

References in corpus (16)

• On the Convergence of Adam and Beyond
• Recovery Guarantees for One-hidden-layer Neural Networks
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• From Averaging to Acceleration, There is Only a Step-size
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Understanding the Role of Momentum in Stochastic Gradient Methods
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Width Provably Matters in Optimization for Deep Linear Neural Networks
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
• Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
• Escaping Saddle Points Faster with Stochastic Momentum
• Generalized Leverage Score Sampling for Neural Networks
• Global Convergence of Second-order Dynamics in Two-layer Neural Networks
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
• Global Convergence of Gradient Descent for Deep Linear Residual Networks
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity