On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

Abstract

In this article we study fully-connected feedforward deep ReLU ANNs with an arbitrarily large number of hidden layers and we prove convergence of the risk of the GD optimization method with random initializations in the training of such ANNs under the assumption that the unnormalized probability density function of the probability distribution of the input data of the considered supervised learning problem is piecewise polynomial, under the assumption that the target function (describing the relationship between input data and the output data) is piecewise polynomial, and under the assumption that the risk function of the considered supervised learning problem admits at least one regular global minimum. In addition, in the special situation of shallow ANNs with just one hidden layer and one-dimensional input we also verify this assumption by proving in the training of such shallow ANNs that for every Lipschitz continuous target function there exists a global minimum in the risk landscape. Finally, in the training of deep ANNs with ReLU activation we also study solutions of gradient flow (GF) differential equations and we prove that every non-divergent GF trajectory converges with a polynomial rate of convergence to a critical point (in the sense of limiting Fréchet subdifferentiability). Our mathematical convergence analysis builds up on ideas from our previous article Eberle et al., on tools from real algebraic geometry such as the concept of semi-algebraic functions and generalized Kurdyka-Lojasiewicz inequalities, on tools from functional analysis such as the Arzelà-Ascoli theorem, on tools from nonsmooth analysis such as the concept of limiting Fréchet subgradients, as well as on the fact that the set of realization functions of shallow ReLU ANNs with fixed architecture forms a closed subset of the set of continuous functions revealed by Petersen et al.

arXiv comment89 pages, 15 figures

References in corpus (25)

• An overview of gradient descent optimization algorithms
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Deep Network Approximation for Smooth Functions
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Strong error analysis for stochastic gradient descent optimization algorithms
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Full error analysis for the training of deep neural networks
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions
• Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
• Cooling down stochastic differential equations: almost sure convergence
• Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks
• Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions
• Stopping Criteria for, and Strong Convergence of, Stochastic Gradient Descent on Bottou-Curtis-Nocedal Functions
• A Dynamical Central Limit Theorem for Shallow Neural Networks

Cited by in corpus (2)

• On the existence of optimal shallow feedforward networks with ReLU activation
• Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes