Strong overall error analysis for the training of artificial neural networks via random initializations

Abstract

Although deep learning based approximation algorithms have been applied very successfully to numerous problems, at the moment the reasons for their performance are not entirely understood from a mathematical point of view. Recently, estimates for the convergence of the overall error have been obtained in the situation of deep supervised learning, but with an extremely slow rate of convergence. In this note we partially improve on these estimates. More specifically, we show that the depth of the neural network only needs to increase much slower in order to obtain the same rate of approximation. The results hold in the case of an arbitrary stochastic optimization algorithm with i.i.d.\ random initializations.

arXiv comment40 pages

References in corpus (11)

• Local Rademacher complexities
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Deep Network Approximation for Smooth Functions
• A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Deep Network Approximation Characterized by Number of Neurons
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Full error analysis for the training of deep neural networks
• Deep neural network approximations for Monte Carlo algorithms