More Data Can Hurt for Linear Regression: Sample-wise Double Descent

Abstract

In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimator. This behavior is implicit in a recent line of theoretical works analyzing "double-descent" phenomenon in linear models. In this note, we isolate and understand this behavior in an extremely simple setting: linear regression with isotropic Gaussian covariates. In particular, this occurs due to an unconventional type of bias-variance tradeoff in the overparameterized regime: the bias decreases with more samples, but variance increases.

References in corpus (5)

• The generalization error of random features regression: Precise asymptotics and double descent curve
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Exact expressions for double descent and implicit regularization via surrogate random design
• On the number of variables to use in principal component regression

Cited by in corpus (22)

• Explaining Neural Scaling Laws
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Optimal Regularization Can Mitigate Double Descent
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization
• What causes the test error? Going beyond bias-variance via ANOVA
• Provable More Data Hurt in High Dimensional Least Squares Estimator
• How more data can hurt: Instability and regularization in next-generation reservoir computing
• On the Universality of the Double Descent Peak in Ridgeless Regression
• On the connection between least squares, regularization, and classical shadows
• The Geometry of Over-parameterized Regression and Adversarial Perturbations
• When and how epochwise double descent happens
• Dimension Independent Generalization Error by Stochastic Gradient Descent
• Risk-Monotonicity in Statistical Learning
• Out-of-Distribution Generalization in Kernel Regression
• Local Linearity and Double Descent in Catastrophic Overfitting
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective
• Mitigating deep double descent by concatenating inputs
• STORM: Foundations of End-to-End Empirical Risk Minimization on the Edge