A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network

Abstract

Incorporating a so-called "momentum" dynamic in gradient descent methods is widely used in neural net training as it has been broadly observed that, at least empirically, it often leads to significantly faster convergence. At the same time, there are very few theoretical guarantees in the literature to explain this apparent acceleration effect. Even for the classical strongly convex quadratic problems, several existing results only show Polyak's momentum has an accelerated linear rate asymptotically. In this paper, we first revisit the quadratic problems and show a non-asymptotic accelerated linear rate of Polyak's momentum. Then, we provably show that Polyak's momentum achieves acceleration for training a one-layer wide ReLU network and a deep linear network, which are perhaps the two most popular canonical models for studying optimization and deep learning in the literature. Prior work Du at al. 2019 and Wu et al. 2019 showed that using vanilla gradient descent, and with an use of over-parameterization, the error decays as $(1-\Theta (\frac{1}{{\kappa }^{\prime }}){)}^t$ after $t$ iterations, where ${\kappa }^{\prime }$ is the condition number of a Gram Matrix. Our result shows that with the appropriate choice of parameters Polyak's momentum has a rate of $(1-\Theta (\frac{1}{\sqrt{{\kappa }^{\prime }}}){)}^t$. For the deep linear network, prior work Hu et al. 2020 showed that vanilla gradient descent has a rate of $(1-\Theta (\frac{1}{\kappa }){)}^t$, where $\kappa$ is the condition number of a data matrix. Our result shows an acceleration rate $(1-\Theta (\frac{1}{\sqrt{\kappa }}){)}^t$ is achievable by Polyak's momentum. All the results in this work are obtained from a modular analysis, which can be of independent interest. This work establishes that momentum does indeed speed up neural net training.

arXiv commentAccepted at ICML 2021

References in corpus (45)

• Decoupled Weight Decay Regularization
• On the Convergence of Adam and Beyond
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Recovery Guarantees for One-hidden-layer Neural Networks
• A Lyapunov Analysis of Momentum Methods in Optimization
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
• Understanding the Acceleration Phenomenon via High-Resolution Differential Equations
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• Depth Creates No Bad Local Minima
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• SGD Learns the Conjugate Kernel Class of the Network
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• From Averaging to Acceleration, There is Only a Step-size
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Global optimality conditions for deep neural networks
• Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Understanding the Role of Momentum in Stochastic Gradient Methods
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Width Provably Matters in Optimization for Deep Linear Neural Networks
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distances
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
• Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
• Learning a Single Neuron with Gradient Methods
• Generalized Leverage Score Sampling for Neural Networks
• Escaping Saddle Points Faster with Stochastic Momentum
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
• Global Convergence of Second-order Dynamics in Two-layer Neural Networks
• Global Convergence of Gradient Descent for Deep Linear Residual Networks
• Parametrized Accelerated Methods Free of Condition Number
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity