A priori generalization error for two-layer ReLU neural network through minimum norm solution
arXiv:1912.03011
Abstract
We focus on estimating \emph{a priori} generalization error of two-layer ReLU neural networks (NNs) trained by mean squared error, which only depends on initial parameters and the target function, through the following research line. We first estimate \emph{a priori} generalization error of finite-width two-layer ReLU NN with constraint of minimal norm solution, which is proved by \cite{zhang2019type} to be an equivalent solution of a linearized (w.r.t. parameter) finite-width two-layer NN. As the width goes to infinity, the linearized NN converges to the NN in Neural Tangent Kernel (NTK) regime \citep{jacot2018neural}. Thus, we can derive the \emph{a priori} generalization error of two-layer ReLU NN in NTK regime. The distance between NN in a NTK regime and a finite-width NN with gradient training is estimated by \cite{arora2019exact}. Based on the results in \cite{arora2019exact}, our work proves an \emph{a priori} generalization error bound of two-layer ReLU NNs. This estimate uses the intrinsic implicit bias of the minimum norm solution without requiring extra regularity in the loss function. This \emph{a priori} estimate also implies that NN does not suffer from curse of dimensionality, and a small generalization error can be achieved without requiring exponentially large number of neurons. In addition the research line proposed in this paper can also be used to study other properties of the finite-width network, such as the posterior generalization error.
arXiv commentThere is a error in this paper that the scale of initialization in this paper is different from the NTK regime. So, the generalization error of the neural network in the NTK regime is baseless
References in corpus (14)
- Understanding deep learning requires rethinking generalization
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- On Exact Computation with an Infinitely Wide Neural Net
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- A Priori Estimates of the Population Risk for Two-layer Neural Networks
- A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
- Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
- Training behavior of deep neural network in frequency domain
- Understanding training and generalization in deep learning by Fourier analysis
- Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
- Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks
- Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
- A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problems