Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing

Abstract

We discuss the prediction accuracy of assumed statistical models in terms of prediction errors for the generalized linear model and penalized maximum likelihood methods. We derive the forms of estimators for the prediction errors, such as $C_p$ criterion, information criteria, and leave-one-out cross validation (LOOCV) error, using the generalized approximate message passing (GAMP) algorithm and replica method. These estimators coincide with each other when the number of model parameters is sufficiently small; however, there is a discrepancy between them in particular in the parameter region where the number of model parameters is larger than the data dimension. In this paper, we review the prediction errors and corresponding estimators, and discuss their differences. In the framework of GAMP, we show that the information criteria can be expressed by using the variance of the estimates. Further, we demonstrate how to approach LOOCV error from the information criteria by utilizing the expression provided by GAMP.

arXiv comment73 pages, 13 figures, accepted Journal of Physics A

References in corpus (22)

• First M87 Event Horizon Telescope Results. I. The Shadow of the Supermassive Black Hole
• Message Passing Algorithms for Compressed Sensing
• Reconciling modern machine learning practice and the bias-variance trade-off
• On the "degrees of freedom" of the lasso
• Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection
• Statistical physics of inference: Thresholds and algorithms
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• A typical reconstruction limit of compressed sensing based on Lp-norm minimization
• The Nishimori line and Bayesian Statistics
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Phase transitions and sample complexity in Bayes-optimal matrix factorization
• Cross validation in LASSO and its acceleration
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Universality of empirical risk minimization
• Equations of States in Statistical Learning for a Nonparametrizable and Regular Case
• Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis
• Evaluation of Generalized Degrees of Freedom for Sparse Estimation by Replica Method
• Estimator of Prediction Error Based on Approximate Message Passing for Penalized Linear Regression
• Posterior covariance information criterion for general loss functions

Cited by in corpus (1)

• Replica Analysis for Ensemble Techniques in Variable Selection