Asymptotics of Ridge (less) Regression under General Source Condition

Abstract

We analyze the prediction error of ridge regression in an asymptotic regime where the sample size and dimension go to infinity at a proportional rate. In particular, we consider the role played by the structure of the true regression parameter. We observe that the case of a general deterministic parameter can be reduced to the case of a random parameter from a structured prior. The latter assumption is a natural adaptation of classic smoothness assumptions in nonparametric regression, which are known as source conditions in the the context of regularization theory for inverse problems. Roughly speaking, we assume the large coefficients of the parameter are in correspondence to the principal components. In this setting a precise characterisation of the test error is obtained, depending on the inputs covariance and regression parameter structure. We illustrate this characterisation in a simplified setting to investigate the influence of the true parameter on optimal regularisation for overparameterized models. We show that interpolation (no regularisation) can be optimal even with bounded signal-to-noise ratio (SNR), provided that the parameter coefficients are larger on high-variance directions of the data, corresponding to a more regular function than posited by the regularization term. This contrasts with previous work considering ridge regression with isotropic prior, in which case interpolation is only optimal in the limit of infinite SNR.

References in corpus (15)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Benign Overfitting in Linear Regression
• On Lazy Training in Differentiable Programming
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• Optimal Regularization Can Mitigate Double Descent
• A Large Dimensional Analysis of Least Squares Support Vector Machines
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• Multiple Descent: Design Your Own Generalization Curve
• On the number of variables to use in principal component regression
• Geometric sensitivity of random matrix results: consequences for shrinkage estimators of covariance and related statistical methods
• Asymptotic errors for convex penalized linear regression beyond Gaussian matrices
• When Does Preconditioning Help or Hurt Generalization?

Cited by in corpus (8)

• Multiple Descent: Design Your Own Generalization Curve
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Kernel regression in high dimensions: Refined analysis beyond double descent
• When Does Preconditioning Help or Hurt Generalization?
• Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
• On the interplay between data structure and loss function in classification problems
• The impact of memory on learning sequence-to-sequence tasks
• Data splitting improves statistical performance in overparametrized regimes