Full error analysis for the training of deep neural networks

Abstract

Deep learning algorithms have been applied very successfully in recent years to a range of problems out of reach for classical solution paradigms. Nevertheless, there is no completely rigorous mathematical error and convergence analysis which explains the success of deep learning algorithms. The error of a deep learning algorithm can in many situations be decomposed into three parts, the approximation error, the generalization error, and the optimization error. In this work we estimate for a certain deep learning algorithm each of these three errors and combine these three error estimates to obtain an overall error analysis for the deep learning algorithm under consideration. In particular, we thereby establish convergence with a suitable convergence speed for the overall error of the deep learning algorithm under consideration. Our convergence speed analysis is far from optimal and the convergence speed that we establish is rather slow, increases exponentially in the dimensions, and, in particular, suffers from the curse of dimensionality. The main contribution of this work is, instead, to provide a full error analysis (i) which covers each of the three different sources of errors usually emerging in deep learning algorithms and (ii) which merges these three sources of errors into one overall error estimate for the considered deep learning algorithm.

arXiv comment53 pages

References in corpus (26)

• Local Rademacher complexities
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Provable approximation properties for deep neural networks
• A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
• Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations
• On the approximation by single hidden layer feedforward neural networks with fixed weights
• Deep Network Approximation Characterized by Number of Neurons
• Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis
• DNN Expression Rate Analysis of High-dimensional PDEs: Application to Option Pricing
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
• Nonlinear Approximation via Compositions
• Approximation capability of two hidden layer feedforward neural networks with fixed weights
• Strong error analysis for stochastic gradient descent optimization algorithms
• A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients
• Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems
• Lower error bounds for the stochastic gradient descent optimization algorithm: Sharp convergence rates for slowly and fast decaying learning rates
• Approximation spaces of deep neural networks
• Non-asymptotic Analysis of Biased Stochastic Approximation Scheme
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• On stochastic gradient Langevin dynamics with dependent data streams: the fully non-convex case
• Exponential Convergence of the Deep Neural Network Approximation for Analytic Functions
• The universal approximation power of finite-width deep ReLU networks
• Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions

Cited by in corpus (15)

• Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
• A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions
• The gap between theory and practice in function approximation with deep neural networks
• Learning the random variables in Monte Carlo simulations with stochastic gradient descent: Machine learning for parametric PDEs and financial derivative pricing
• Partition of unity networks: deep hp-approximation
• Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data
• High-dimensional approximation spaces of artificial neural networks and applications to partial differential equations
• A Neural Solver for Variational Problems on CAD Geometries with Application to Electric Machine Simulation
• Convergence rates for gradient descent in the training of overparameterized artificial neural networks with piecewise affine activation
• On the approximation of functions by tanh neural networks
• Strong overall error analysis for the training of artificial neural networks via random initializations
• Polynomial-Spline Neural Networks with Exact Integrals
• Multilevel Monte Carlo learning