A Priori Estimates of the Population Risk for Residual Networks

Abstract

Optimal a priori estimates are derived for the population risk, also known as the generalization error, of a regularized residual network model. An important part of the regularized model is the usage of a new path norm, called the weighted path norm, as the regularization term. The weighted path norm treats the skip connections and the nonlinearities differently so that paths with more nonlinearities are regularized by larger weights. The error estimates are a priori in the sense that the estimates depend only on the target function, not on the parameters obtained in the training process. The estimates are optimal, in a high dimensional setting, in the sense that both the bound for the approximation and estimation errors are comparable to the Monte Carlo error rates. A crucial step in the proof is to establish an optimal bound for the Rademacher complexity of the residual networks. Comparisons are made with existing norm-based generalization error bounds.

References in corpus (12)

• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Stronger generalization bounds for deep nets via a compression approach
• Approximation and Estimation for High-Dimensional Deep Learning Networks
• Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks
• Capacity Control of ReLU Neural Networks by Basis-path Norm

Cited by in corpus (25)

• A Review on Deep Learning in Medical Image Reconstruction
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Optimal Approximation Rate of ReLU Networks in terms of Width and Depth
• SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations
• Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• A Selective Overview of Deep Learning
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• Reproducing Activation Function for Deep Learning
• Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
• Generalization and Memorization: The Bias Potential Model
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
• The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
• The Barron Space and the Flow-induced Function Spaces for Neural Network Models
• Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)
• A Mathematical Framework for Learning Probability Distributions
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networks
• Dimension Independent Generalization Error by Stochastic Gradient Descent
• Learning the mapping $\mathbf{x}\mapsto {\sum }_{i=1}^d{x}_i^2$: the cost of finding the needle in a haystack