Towards an Understanding of Benign Overfitting in Neural Networks

Abstract

Modern machine learning models often employ a huge number of parameters and are typically optimized to have zero training loss; yet surprisingly, they possess near-optimal prediction performance, contradicting classical learning theory. We examine how these benign overfitting phenomena occur in a two-layer neural network setting where sample covariates are corrupted with noise. We address the high dimensional regime, where the data dimension $d$ grows with the number $n$ of data points. Our analysis combines an upper bound on the bias with matching upper and lower bounds on the variance of the interpolator (an estimator that interpolates the data). These results indicate that the excess learning risk of the interpolator decays under mild conditions. We further show that it is possible for the two-layer ReLU network interpolator to achieve a near minimax-optimal learning rate, which to our knowledge is the first generalization result for such networks. Finally, our theory predicts that the excess learning risk starts to increase once the number of parameters $s$ grows beyond $O(n^2)$, matching recent empirical findings.

References in corpus (14)

• Reconciling modern machine learning practice and the bias-variance trade-off
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guarantees
• Optimal Regularization Can Mitigate Double Descent
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
• Exact expressions for double descent and implicit regularization via surrogate random design
• Multiple Descent: Design Your Own Generalization Curve
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures
• Interpolating Predictors in High-Dimensional Factor Regression
• Ivanov-Regularised Least-Squares Estimators over Large RKHSs and Their Interpolation Spaces
• Benign Overfitting and Noisy Features
• Risk of the Least Squares Minimum Norm Estimator under the Spike Covariance Model

Cited by in corpus (4)

• On the Double Descent of Random Features Models Trained with SGD
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks
• Harmless interpolation in regression and classification with structured features
• The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks