Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
arXiv:2111.14039
Abstract
In this paper, we study the generalization performance of global minima for implementing empirical risk minimization (ERM) on over-parameterized deep ReLU nets. Using a novel deepening scheme for deep ReLU nets, we rigorously prove that there exist perfect global minima achieving almost optimal generalization error bounds for numerous types of data under mild conditions. Since over-parameterization is crucial to guarantee that the global minima of ERM on deep ReLU nets can be realized by the widely used stochastic gradient descent (SGD) algorithm, our results indeed fill a gap between optimization and generalization.
arXiv comment15 pages, 3 figures
References in corpus (15)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Understanding deep learning requires rethinking generalization
- A Convergence Theory for Deep Learning via Over-Parameterization
- Benign Overfitting in Linear Regression
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Provable approximation properties for deep neural networks
- Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
- On the Expressive Power of Deep Neural Networks
- Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variation
- Over-parametrized deep neural networks do not generalize well
- Norm-based generalisation bounds for multi-class convolutional neural networks
- Kernel Interpolation of High Dimensional Scattered Data