Approximation and Estimation for High-Dimensional Deep Learning Networks

Abstract

It has been experimentally observed in recent years that multi-layer artificial neural networks have a surprising ability to generalize, even when trained with far more parameters than observations. Is there a theoretical basis for this? The best available bounds on their metric entropy and associated complexity measures are essentially linear in the number of parameters, which is inadequate to explain this phenomenon. Here we examine the statistical risk (mean squared predictive error) of multi-layer networks with ${\ell }^1$-type controls on their parameters and with ramp activation functions (also called lower-rectified linear units). In this setting, the risk is shown to be upper bounded by $[(L^3\log d)/n{]}^{1/2}$, where $d$ is the input dimension to each layer, $L$ is the number of layers, and $n$ is the sample size. In this way, the input dimension can be much larger than the sample size and the estimator can still be accurate, provided the target function has such ${\ell }^1$ controls and that the sample size is at least moderately large compared to $L^3\log d$. The heart of the analysis is the development of a sampling strategy that demonstrates the accuracy of a sparse covering of deep ramp networks. Lower bounds show that the identified risk is close to being optimal.

References in corpus (8)

• Approximation and learning by greedy algorithms
• Spectrally-normalized margin bounds for neural networks
• Optimal approximation of continuous functions by very deep ReLU networks
• Stronger generalization bounds for deep nets via a compression approach
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Exponential Screening and optimal rates of sparse estimation
• Norm-Based Capacity Control in Neural Networks
• Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks

Cited by in corpus (19)

• Deep Network Approximation for Smooth Functions
• Neural Network Approximation: Three Hidden Layers Are Enough
• The Modern Mathematics of Deep Learning
• A Theoretical Analysis of Deep Q-Learning
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• A Priori Estimates of the Population Risk for Residual Networks
• Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variation
• Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
• Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality
• Formalizing Generalization and Robustness of Neural Networks to Weight Perturbations
• On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning
• Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
• Optimization Landscapes of Wide Deep Neural Networks Are Benign
• Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems
• Robust Estimation in Finite Mixture Models
• Distance Metric-Based Learning with Interpolated Latent Features for Location Classification in Endoscopy Image and Video
• Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning