Cross validation in LASSO and its acceleration

Abstract

We investigate leave-one-out cross validation (CV) as a determinator of the weight of the penalty term in the least absolute shrinkage and selection operator (LASSO). First, on the basis of the message passing algorithm and a perturbative discussion assuming that the number of observations is sufficiently large, we provide simple formulas for approximately assessing two types of CV errors, which enable us to significantly reduce the necessary cost of computation. These formulas also provide a simple connection of the CV errors to the residual sums of squares between the reconstructed and the given measurements. Second, on the basis of this finding, we analytically evaluate the CV errors when the design matrix is given as a simple random matrix in the large size limit by using the replica method. Finally, these results are compared with those of numerical simulations on finite-size systems and are confirmed to be correct. We also apply the simple formulas of the first type of CV error to an actual dataset of the supernovae.

32 pages, 7 figures

References in corpus (2)

• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• Variable Selection for Modeling the Absolute Magnitude at Maximum of Type Ia Supernovae

Cited by in corpus (10)

• Imaging the Schwarzschild-radius-scale Structure of M87 with the Event Horizon Telescope using Sparse Modeling
• Super-resolution Full Polarimetric Imaging for Radio Interferometry with Sparse Modeling
• Sparse Modeling in Quantum Many-Body Problems
• Real-space analysis of scanning tunneling microscopy topography datasets using sparse modeling approach
• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing
• Cross validation in sparse linear regression with piecewise continuous nonconvex penalties and its acceleration
• Evaluation of Generalized Degrees of Freedom for Sparse Estimation by Replica Method
• Role of Bootstrap Averaging in Generalized Approximate Message Passing
• Reconstructing Sparse Signals via Greedy Monte-Carlo Search
• Automatic Hyperparameter Tuning in Sparse Matrix Factorization