On the "degrees of freedom" of the lasso

Abstract

We study the effective degrees of freedom of the lasso in the framework of Stein's unbiased risk estimation (SURE). We show that the number of nonzero coefficients is an unbiased estimate for the degrees of freedom of the lasso--a conclusion that requires no special assumption on the predictors. In addition, the unbiased estimator is shown to be asymptotically consistent. With these results on hand, various model selection criteria--$C_p$, AIC and BIC--are available, which, along with the LARS algorithm, provide a principled and efficient approach to obtaining the optimal lasso fit with the computational effort of a single ordinary least-squares fit.

arXiv commentPublished in at http://dx.doi.org/10.1214/009053607000000127 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (12)

• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Nonconcave penalized likelihood with a diverging number of parameters
• Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery

Cited by in corpus (146)

• On the adaptive elastic-net with a diverging number of parameters
• The composite absolute penalties family for grouped and hierarchical variable selection
• A significance test for the lasso
• The solution path of the generalized lasso
• Estimating treatment effect heterogeneity in randomized program evaluation
• Generalized Network Psychometrics: Combining Network and Latent Variable Models
• A lasso for hierarchical interactions
• Degrees of freedom in lasso problems
• Sparse least trimmed squares regression for analyzing high-dimensional large data sets
• Regularized Matrix Regression
• A proximal iteration for deconvolving Poisson noisy images using sparse representations
• Least angle and ${\ell }_1$ penalized regression: A review
• Multimodal Task-Driven Dictionary Learning for Image Classification
• Functional additive regression
• Estimation for High-Dimensional Linear Mixed-Effects Models Using ${\ell }_1$-Penalization
• Machine Learning Estimation of Heterogeneous Causal Effects: Empirical Monte Carlo Evidence
• Sparse logistic principal components analysis for binary data
• A new perspective on least squares under convex constraint
• GLMMLasso: An Algorithm for High-Dimensional Generalized Linear Mixed Models Using L1-Penalization
• Tensor Analysis and Fusion of Multimodal Brain Images
• Penalized model-based clustering with cluster-specific diagonal covariance matrices and grouped variables
• Consistent selection of tuning parameters via variable selection stability
• Efficiency for Regularization Parameter Selection in Penalized Likelihood Estimation of Misspecified Models
• Statistical Significance of the Netflix Challenge
• Bayesian model choice and information criteria in sparse generalized linear models
• Penalized variable selection procedure for Cox models with semiparametric relative risk
• Effective Degrees of Freedom: A Flawed Metaphor
• The Lasso with general Gaussian designs with applications to hypothesis testing
• A Primal Dual Active Set Algorithm with Continuation for Compressed Sensing
• Regularized 3D functional regression for brain image data via Haar wavelets
• An Ordered Lasso and Sparse Time-Lagged Regression
• MIMO Multiway Relaying with Pairwise Data Exchange: A Degrees of Freedom Perspective
• Prediction error of cross-validated Lasso
• Adaptive Filter for Automatic Identification of Multiple Faults in a Noisy OTDR Profile
• Sparsity with sign-coherent groups of variables via the cooperative-Lasso
• Comment: Boosting Algorithms: Regularization, Prediction and Model Fitting
• Local Behavior of Sparse Analysis Regularization: Applications to Risk Estimation
• Aiming Perfectly in the Dark - Blind Interference Alignment through Staggered Antenna Switching
• Implicit differentiation of Lasso-type models for hyperparameter optimization
• A Path Algorithm for Constrained Estimation
• Harmonization of diffusion MRI datasets with adaptive dictionary learning
• Iteratively reweighted adaptive lasso for conditional heteroscedastic time series with applications to AR-ARCH type processes
• Effective New Methods for Automated Parameter Selection in Regularized Inverse Problems
• Asymptotic Analysis of LASSOs Solution Path with Implications for Approximate Message Passing
• Smoothed Bernstein Online Aggregation for Day-Ahead Electricity Demand Forecasting
• Non-Concave Penalization in Linear Mixed-Effects Models and Regularized Selection of Fixed Effects
• A scalable estimate of the extra-sample prediction error via approximate leave-one-out
• On Improved Loss Estimation for Shrinkage Estimators
• Identifying Network Ties from Panel Data: Theory and an Application to Tax Competition
• Regularization with the Smooth-Lasso procedure
• De-Biasing The Lasso With Degrees-of-Freedom Adjustment
• A neural network model for solvency calculations in life insurance
• High-Dimensional Linear Regression via Implicit Regularization
• Rebar: Reinforcing a Matching Estimator with Predictions from High-Dimensional Covariates
• Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions
• Learning networks from high dimensional binary data: An application to genomic instability data
• A Generalized Least Squares Matrix Decomposition
• Phase retrieval from single biomolecule diffraction pattern
• On the Sensitivity of the Lasso to the Number of Predictor Variables
• Greedy algorithms for prediction
• Variable selection and estimation for semi-parametric multiple-index models
• regsem: Regularized Structural Equation Modeling
• Robust Sparse Analysis Regularization
• Degrees of freedom for nonlinear least squares estimation
• Approximate Leave-One-Out for High-Dimensional Non-Differentiable Learning Problems
• On the Degrees of Freedom of the Symmetric Multi-Relay MIMO Y Channel
• The restricted consistency property of leave-$n_v$-out cross-validation for high-dimensional variable selection
• SURE Information Criteria for Large Covariance Matrix Estimation and Their Asymptotic Properties
• A study on tuning parameter selection for the high-dimensional lasso
• Efficient algorithm to select tuning parameters in sparse regression modeling with regularization
• Algorithms for Fitting the Constrained Lasso
• A Century of Economic Policy Uncertainty Through the French-Canadian Lens
• AIC for Non-concave Penalized Likelihood Method
• Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis
• Sparse estimation via nonconcave penalized likelihood in a factor analysis model
• Parameter Choices for Sparse Regularization with the ${\ell }_1$ Norm
• A Survey of Tuning Parameter Selection for High-dimensional Regression
• Quantile universal threshold for model selection
• Inferring Signaling Pathways with Probabilistic Programming
• The Loss Rank Criterion for Variable Selection in Linear Regression Analysis
• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing
• Real Interference Alignment
• Probabilistic Reduced-Order Modeling for Stochastic Partial Differential Equations
• Boosted nonparametric hazards with time-dependent covariates
• Multinomial Multiple Correspondence Analysis
• Feature-specific inference for penalized regression using local false discovery rates
• The Degrees of Freedom of Partly Smooth Regularizers
• Penalized Likelihood Regression in Reproducing Kernel Hilbert Spaces with Randomized Covariate Data
• Degrees of Freedom and Model Search
• A New DoF Upper Bound and Its Achievability for $K$-User MIMO Y Channels
• Regression shrinkage and grouping of highly correlated predictors with HORSES
• Mixture of von Mises-Fisher distribution with sparse prototypes
• Evaluation of Generalized Degrees of Freedom for Sparse Estimation by Replica Method
• Multiple-Instance Logistic Regression with LASSO Penalty
• Mixture Components Inference for Sparse Regression: Introduction and Application for Estimation of Neuronal Signal from fMRI BOLD
• REMI: Regression with marginal information and its application in genome-wide association studies
• Adaptive Higher-order Spectral Estimators
• Investigating Competition in Financial Markets: A Sparse Autologistic Model for Dynamic Network Data
• Degrees of Freedom in Deep Neural Networks
• Adding smoothing splines to the SAM model improves stock assessment
• Estimator of Prediction Error Based on Approximate Message Passing for Penalized Linear Regression
• Variance Estimation Using Refitted Cross-validation in Ultrahigh Dimensional Regression
• Path Thresholding: Asymptotically Tuning-Free High-Dimensional Sparse Regression
• The Degrees of Freedom of the Group Lasso
• Credit scoring using neural networks and SURE posterior probability calibration
• A Generic Path Algorithm for Regularized Statistical Estimation
• Blockwise and coordinatewise thresholding to combine tests of different natures in modern ANOVA
• Marginal false discovery rates for penalized regression models
• Estimating the Rate Constant from Biosensor Data via an Adaptive Variational Bayesian Approach
• Whole-brain Prediction Analysis with GraphNet
• Revisiting minimum description length complexity in overparameterized models
• Semi-supervised linear regression: enhancing efficiency and robustness in high dimensions
• Automatic Hyperparameter Tuning in Sparse Matrix Factorization
• Constrained maximum likelihood estimation of clusterwise linear regression models with unknown number of components
• A Note on the Forward-Douglas--Rachford Splitting for Monotone Inclusion and Convex Optimization
• From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation
• Sparse Correspondence Analysis for Contingency Tables
• Spatial point processes intensity estimation with a diverging number of covariates
• Quantile universal threshold: model selection at the detection edge for high-dimensional linear regression
• Asymmetric Degrees of Freedom of the Full-Duplex MIMO 3-Way Channel with Unicast and Broadcast Messages
• Weak Signal Identification and Inference in Penalized Model Selection
• Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?
• Estimation of oblique structure via penalized likelihood factor analysis
• Graph estimation with joint additive models
• Independent screening for single-index hazard rate models with ultra-high dimensional features
• Partial Correlation Estimation by Joint Sparse Regression Models
• Capacity of a Class of Symmetric SIMO Gaussian Interference Channels within O(1)
• Estimating the Penalty Level of ${\ell }_1$-minimization via Two Gaussian Approximation Methods
• Generating Probabilities From Numerical Weather Forecasts by Logistic Regression
• Spatially Clustered Regression
• Computing the degrees of freedom of rank-regularized estimators and cousins
• On the Consistency of the Bias Correction Term of the AIC for the Non-Concave Penalized Likelihood Method
• yaglm: a Python package for fitting and tuning generalized linear models that supports structured, adaptive and non-convex penalties
• Smooth blockwise iterative thresholding: a smooth fixed point estimator based on the likelihood's block gradient
• Recovering lost and absent information in temporal networks
• Simultaneous disease mapping and hot spot detection with application to childhood obesity surveillance from electronic health records
• Regularization for the Approximation of 2D Set of Points via the Length of the Curve
• Assessment of Case Influence in the Lasso with a Case-weight Adjusted Solution Path
• A non-convex regularization approach for stable estimation of loss development factors
• Weak convergence of the regularization path in penalized M-estimation
• Estimation and Selection Properties of the LAD Fused Lasso Signal Approximator
• Model-based clustering of multivariate binary data with dimension reduction
• Bridging Information Criteria and Parameter Shrinkage for Model Selection
• Modified Cross-Validation for Penalized High-Dimensional Linear Regression Models
• Prediction Weighted Maximum Frequency Selection
• gLOP: the global and Local Penalty for Capturing Predictive Heterogeneity