Dimensionality reduction, regularization, and generalization in overparameterized regressions

Abstract

Overparameterization in deep learning is powerful: Very large models fit the training data perfectly and yet often generalize well. This realization brought back the study of linear models for regression, including ordinary least squares (OLS), which, like deep learning, shows a "double-descent" behavior: (1) The risk (expected out-of-sample prediction error) can grow arbitrarily when the number of parameters $p$ approaches the number of samples $n$, and (2) the risk decreases with $p$ for $p>n$, sometimes achieving a lower value than the lowest risk for $p<n$. The divergence of the risk for OLS can be avoided with regularization. In this work, we show that for some data models it can also be avoided with a PCA-based dimensionality reduction (PCA-OLS, also known as principal component regression). We provide non-asymptotic bounds for the risk of PCA-OLS by considering the alignments of the population and empirical principal components. We show that dimensionality reduction improves robustness while OLS is arbitrarily susceptible to adversarial attacks, particularly in the overparameterized regime. We compare PCA-OLS theoretically and empirically with a wide range of projection-based methods, including random projections, partial least squares (PLS), and certain classes of linear two-layer neural networks. These comparisons are made for different data generation models to assess the sensitivity to signal-to-noise and the alignment of regression coefficients with the features. We find that methods in which the projection depends on the training data can outperform methods where the projections are chosen independently of the training data, even those with oracle knowledge of population quantities, another seemingly paradoxical phenomenon that has been identified previously. This suggests that overparameterization may not be necessary for good generalization.

References in corpus (14)

• Poisoning Attacks against Support Vector Machines
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• Normal approximation and concentration of spectral projectors of sample covariance
• Optimal Regularization Can Mitigate Double Descent
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• What causes the test error? Going beyond bias-variance via ANOVA
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Kernel regression in high dimensions: Refined analysis beyond double descent
• How to reduce dimension with PCA and random projections?
• Overfitting Can Be Harmless for Basis Pursuit, But Only to a Degree
• Provable More Data Hurt in High Dimensional Least Squares Estimator
• Optimal Feature Manipulation Attacks Against Linear Regression

Cited by in corpus (2)

• Support vector machines and linear regression coincide with very high-dimensional features
• The High-Dimensional Asymptotics of Principal Component Regression