A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions

Abstract

Gradient descent optimization algorithms are the standard ingredients that are used to train artificial neural networks (ANNs). Even though a huge number of numerical simulations indicate that gradient descent optimization methods do indeed convergence in the training of ANNs, until today there is no rigorous theoretical analysis which proves (or disproves) this conjecture. In particular, even in the case of the most basic variant of gradient descent optimization algorithms, the plain vanilla gradient descent method, it remains an open problem to prove or disprove the conjecture that gradient descent converges in the training of ANNs. In this article we solve this problem in the special situation where the target function under consideration is a constant function. More specifically, in the case of constant target functions we prove in the training of rectified fully-connected feedforward ANNs with one-hidden layer that the risk function of the gradient descent method does indeed converge to zero. Our mathematical analysis strongly exploits the property that the rectifier function is the activation function used in the considered ANNs. A key contribution of this work is to explicitly specify a Lyapunov function for the gradient flow system of the ANN parameters. This Lyapunov function is the central tool in our convergence proof of the gradient descent method.

arXiv comment23 pages

References in corpus (17)

• An overview of gradient descent optimization algorithms
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Dying ReLU and Initialization: Theory and Numerical Examples
• Collapse of Deep and Narrow Neural Nets
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Strong error analysis for stochastic gradient descent optimization algorithms
• Full error analysis for the training of deep neural networks
• Non-convergence of stochastic gradient descent in the training of deep neural networks
• Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning rates
• A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions
• Nonasymptotic analysis of Stochastic Gradient Hamiltonian Monte Carlo under local conditions for nonconvex optimization
• Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions

Cited by in corpus (9)

• Full error analysis for the training of deep neural networks
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
• A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions
• Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
• Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation
• Gradient descent provably escapes saddle points in the training of shallow ReLU networks
• Convergence rates for gradient descent in the training of overparameterized artificial neural networks with piecewise affine activation
• A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions