A Priori Estimates of the Population Risk for Two-layer Neural Networks

Abstract

New estimates for the population risk are established for two-layer neural networks. These estimates are nearly optimal in the sense that the error rates scale in the same way as the Monte Carlo error rates. They are equally effective in the over-parametrized regime when the network size is much larger than the size of the dataset. These new estimates are a priori in nature in the sense that the bounds depend only on some norms of the underlying functions to be fitted, not the parameters in the model, in contrast with most existing results which are a posteriori in nature. Using these a priori estimates, we provide a perspective for understanding why two-layer neural networks perform better than the related kernel methods.

arXiv commentPublished version

References in corpus (11)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Norm-Based Capacity Control in Neural Networks
• SGD Learns the Conjugate Kernel Class of the Network
• On the loss landscape of a class of deep neural networks with no bad local valleys
• A Priori Estimates of the Population Risk for Residual Networks
• Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks
• Topology and Geometry of Half-Rectified Network Optimization
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections

Cited by in corpus (50)

• Deep Potentials for Materials Science
• Deep Network Approximation for Smooth Functions
• A Review on Deep Learning in Medical Image Reconstruction
• Deep learning observables in computational fluid dynamics
• Neural Network Approximation: Three Hidden Layers Are Enough
• The Modern Mathematics of Deep Learning
• The Finite Neuron Method and Convergence Analysis
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• Supervised learning from noisy observations: Combining machine-learning techniques with data assimilation
• SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations
• Machine Learning for Prediction with Missing Dynamics
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• Machine Learning from a Continuous Viewpoint
• A Priori Estimates of the Population Risk for Residual Networks
• An augmented Lagrangian deep learning method for variational problems with essential boundary conditions
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Representation formulas and pointwise properties for Barron functions
• Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variation
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• A Priori Analysis of Stable Neural Network Solutions to Numerical PDEs
• Reproducing Activation Function for Deep Learning
• How Powerful are Shallow Neural Networks with Bandlimited Random Weights?
• Generalization and Memorization: The Bias Potential Model
• Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
• Conditional Density Estimation, Latent Variable Discovery and Optimal Transport
• Solving PDEs on Unknown Manifolds with Machine Learning
• The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
• The Barron Space and the Flow-induced Function Spaces for Neural Network Models
• A priori estimates for classification problems using neural networks
• Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
• Universality and approximation bounds for echo state networks with random weights
• Spectral Barron space for deep neural network approximation
• A brief note on understanding neural networks as Gaussian processes
• Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methods
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networks
• A priori generalization error for two-layer ReLU neural network through minimum norm solution
• Piecewise Linear Functions Representable with Infinite Width Shallow ReLU Neural Networks
• Making Method of Moments Great Again? -- How can GANs learn distributions
• Strong overall error analysis for the training of artificial neural networks via random initializations
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• Stationary Density Estimation of Itô Diffusions Using Deep Learning