Dimensionality reduction, regularization, and generalization in overparameterized regressions

Abstract

Overparameterization in deep learning is powerful: Very large models fit the training data perfectly and yet often generalize well. This realization brought back the study of linear models for regression, including ordinary least squares (OLS), which, like deep learning, shows a "double-descent" behavior: (1) The risk (expected out-of-sample prediction error) can grow arbitrarily when the number of parameters $p$ approaches the number of samples $n$, and (2) the risk decreases with $p$ for $p>n$, sometimes achieving a lower value than the lowest risk for $p<n$. The divergence of the risk for OLS can be avoided with regularization. In this work, we show that for some data models it can also be avoided with a PCA-based dimensionality reduction (PCA-OLS, also known as principal component regression). We provide non-asymptotic bounds for the risk of PCA-OLS by considering the alignments of the population and empirical principal components. We show that dimensionality reduction improves robustness while OLS is arbitrarily susceptible to adversarial attacks, particularly in the overparameterized regime. We compare PCA-OLS theoretically and empirically with a wide range of projection-based methods, including random projections, partial least squares (PLS), and certain classes of linear two-layer neural networks. These comparisons are made for different data generation models to assess the sensitivity to signal-to-noise and the alignment of regression coefficients with the features. We find that methods in which the projection depends on the training data can outperform methods where the projections are chosen independently of the training data, even those with oracle knowledge of population quantities, another seemingly paradoxical phenomenon that has been identified previously. This suggests that overparameterization may not be necessary for good generalization.

References in corpus (27)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Poisoning Attacks against Support Vector Machines
• Benign Overfitting in Linear Regression
• The Cannon: A data-driven approach to stellar label determination
• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Two models of double descent for weak features
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• Normal approximation and concentration of spectral projectors of sample covariance
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• Optimal Regularization Can Mitigate Double Descent
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• Are Generative Classifiers More Robust to Adversarial Attacks?
• On the number of variables to use in principal component regression
• What causes the test error? Going beyond bias-variance via ANOVA
• On the Adversarial Robustness of Subspace Learning
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Kernel regression in high dimensions: Refined analysis beyond double descent
• Overfitting Can Be Harmless for Basis Pursuit, But Only to a Degree
• How to reduce dimension with PCA and random projections?
• Provable More Data Hurt in High Dimensional Least Squares Estimator
• Optimal Feature Manipulation Attacks Against Linear Regression
• A note on the prediction error of principal component regression

Cited by in corpus (2)

• Support vector machines and linear regression coincide with very high-dimensional features
• The High-Dimensional Asymptotics of Principal Component Regression