Estimator of Prediction Error Based on Approximate Message Passing for Penalized Linear Regression

Abstract

We propose an estimator of prediction error using an approximate message passing (AMP) algorithm that can be applied to a broad range of sparse penalties. Following Stein's lemma, the estimator of the generalized degrees of freedom, which is a key quantity for the construction of the estimator of the prediction error, is calculated at the AMP fixed point. The resulting form of the AMP-based estimator does not depend on the penalty function, and its value can be further improved by considering the correlation between predictors. The proposed estimator is asymptotically unbiased when the components of the predictors and response variables are independently generated according to a Gaussian distribution. We examine the behaviour of the estimator for real data under nonconvex sparse penalties, where Akaike's information criterion does not correspond to an unbiased estimator of the prediction error. The model selected by the proposed estimator is close to that which minimizes the true prediction error.

References in corpus (5)

• Nearly unbiased variable selection under minimax concave penalty
• On the "degrees of freedom" of the lasso
• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis
• Evaluation of Generalized Degrees of Freedom for Sparse Estimation by Replica Method

Cited by in corpus (1)

• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing