The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks

Abstract

The recent success of neural network models has shone light on a rather surprising statistical phenomenon: statistical models that perfectly fit noisy data can generalize well to unseen test data. Understanding this phenomenon of $\text{benign overfitting}$ has attracted intense theoretical and empirical study. In this paper, we consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk when the covariates satisfy sub-Gaussianity and anti-concentration properties, and the noise is independent and sub-Gaussian. By leveraging recent results that characterize the implicit bias of this estimator, our bounds emphasize the role of both the quality of the initialization as well as the properties of the data covariance matrix in achieving low excess risk.

arXiv commentAccepted for publication at JMLR

References in corpus (12)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Benign Overfitting in Linear Regression
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Benign overfitting in ridge regression
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
• Towards an Understanding of Benign Overfitting in Neural Networks
• Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting
• Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm