On the number of variables to use in principal component regression

Abstract

We study least squares linear regression over $N$ uncorrelated Gaussian features that are selected in order of decreasing variance. When the number of selected features $p$ is at most the sample size $n$, the estimator under consideration coincides with the principal component regression estimator; when $p>n$, the estimator is the least ${\ell }_2$ norm solution over the selected features. We give an average-case analysis of the out-of-sample prediction error as $p,n,N\to \infty$ with $p/N\to \alpha$ and $n/N\to \beta$, for some constants $\alpha \in [0,1]$ and $\beta \in (0,1)$. In this average-case setting, the prediction error exhibits a "double descent" shape as a function of $p$. We also establish conditions under which the minimum risk is achieved in the interpolating ($p>n$) regime.

Cited by in corpus (14)

• Benign overfitting in ridge regression
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Optimal Regularization Can Mitigate Double Descent
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Multiple Descent: Design Your Own Generalization Curve
• Asymptotics of Ridge (less) Regression under General Source Condition
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• When Does Preconditioning Help or Hurt Generalization?
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Dimensionality reduction, regularization, and generalization in overparameterized regressions
• Provable More Data Hurt in High Dimensional Least Squares Estimator
• The Benefits of Implicit Regularization from SGD in Least Squares Problems
• Subspace Fitting Meets Regression: The Effects of Supervision and Orthonormality Constraints on Double Descent of Generalization Errors
• The Geometry of Over-parameterized Regression and Adversarial Perturbations