Regularization in High-Dimensional Regression and Classification via Random Matrix Theory

Abstract

We study general singular value shrinkage estimators in high-dimensional regression and classification, when the number of features and the sample size both grow proportionally to infinity. We allow models with general covariance matrices that include a large class of data generating distributions. As far as the implications of our results are concerned, we find exact asymptotic formulas for both the training and test errors in regression models fitted by gradient descent, which provides theoretical insights for early stopping as a regularization method. In addition, we propose a numerical method based on the empirical spectra of covariance matrices for the optimal eigenvalue shrinkage classifier in linear discriminant analysis. Finally, we derive optimal estimators for the dense mean vectors of high-dimensional distributions. Throughout our analysis we rely on recent advances in random matrix theory and develop further results of independent mathematical interest.

References in corpus (10)

• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• High-dimensional classification using features annealed independence rules
• The generalization error of random features regression: Precise asymptotics and double descent curve
• A modern maximum-likelihood theory for high-dimensional logistic regression
• High Dimensional Statistical Inference and Random Matrices
• Minimax risk of matrix denoising by singular value thresholding
• Efficient Computation of Limit Spectra of Sample Covariance Matrices
• Nonparametric estimate of spectral density functions of sample covariance matrices: A first step
• A Continuous-Time View of Early Stopping for Least Squares
• Sharp Asymptotics and Optimal Performance for Inference in Binary Models

Cited by in corpus (8)

• A Model of Double Descent for High-dimensional Binary Linear Classification
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
• Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• A Precise Performance Analysis of Support Vector Regression
• Asymptotic Risk of Overparameterized Likelihood Models: Double Descent Theory for Deep Neural Networks
• Shrinkage Estimation of Functions of Large Noisy Symmetric Matrices