Provable More Data Hurt in High Dimensional Least Squares Estimator
arXiv:2008.06296
Abstract
This paper investigates the finite-sample prediction risk of the high-dimensional least squares estimator. We derive the central limit theorem for the prediction risk when both the sample size and the number of features tend to infinity. Furthermore, the finite-sample distribution and the confidence interval of the prediction risk are provided. Our theoretical results demonstrate the sample-wise nonmonotonicity of the prediction risk and confirm "more data hurt" phenomenon.
References in corpus (12)
- The generalization error of random features regression: Precise asymptotics and double descent curve
- The jamming transition as a paradigm to understand the loss landscape of deep neural networks
- The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
- More Data Can Hurt for Linear Regression: Sample-wise Double Descent
- Exact expressions for double descent and implicit regularization via surrogate random design
- Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)
- The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
- On the number of variables to use in principal component regression
- More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models
- Asymptotic errors for convex penalized linear regression beyond Gaussian matrices
- Precise Tradeoffs in Adversarial Training for Linear Regression
- Subspace Fitting Meets Regression: The Effects of Supervision and Orthonormality Constraints on Double Descent of Generalization Errors