The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks

Abstract

The recent success of neural network models has shone light on a rather surprising statistical phenomenon: statistical models that perfectly fit noisy data can generalize well to unseen test data. Understanding this phenomenon of $\text{benign overfitting}$ has attracted intense theoretical and empirical study. In this paper, we consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk when the covariates satisfy sub-Gaussianity and anti-concentration properties, and the noise is independent and sub-Gaussian. By leveraging recent results that characterize the implicit bias of this estimator, our bounds emphasize the role of both the quality of the initialization as well as the properties of the data covariance matrix in achieving low excess risk.

Accepted for publication at JMLR

References in corpus (6)

• Benign overfitting in ridge regression
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• Towards an Understanding of Benign Overfitting in Neural Networks
• Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfitting
• Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm