A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions

Abstract

In this article we study the stochastic gradient descent (SGD) optimization method in the training of fully-connected feedforward artificial neural networks with ReLU activation. The main result of this work proves that the risk of the SGD process converges to zero if the target function under consideration is constant. In the established convergence result the considered artificial neural networks consist of one input layer, one hidden layer, and one output layer (with $d\in \mathbb{N}$ neurons on the input layer, $H\in \mathbb{N}$ neurons on the hidden layer, and one neuron on the output layer). The learning rates of the SGD process are assumed to be sufficiently small and the input data used in the SGD process to train the artificial neural networks is assumed to be independent and identically distributed.

arXiv comment29 pages

References in corpus (25)

• An overview of gradient descent optimization algorithms
• A Convergence Theory for Deep Learning via Over-Parameterization
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Dying ReLU and Initialization: Theory and Numerical Examples
• Collapse of Deep and Narrow Neural Nets
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Solving the Kolmogorov PDE by means of deep learning
• Strong error analysis for stochastic gradient descent optimization algorithms
• Full error analysis for the training of deep neural networks
• Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions
• Non-convergence of stochastic gradient descent in the training of deep neural networks
• Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning rates
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• Non-asymptotic Analysis of Biased Stochastic Approximation Scheme
• Nonasymptotic analysis of Stochastic Gradient Hamiltonian Monte Carlo under local conditions for nonconvex optimization
• Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
• Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms
• Convergence rates for gradient descent in the training of overparameterized artificial neural networks with piecewise affine activation
• Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation

Cited by in corpus (5)

• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
• Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation
• A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions