On the Distribution of Penalized Maximum Likelihood Estimators: The LASSO, SCAD, and Thresholding

Abstract

We study the distributions of the LASSO, SCAD, and thresholding estimators, in finite samples and in the large-sample limit. The asymptotic distributions are derived for both the case where the estimators are tuned to perform consistent model selection and for the case where the estimators are tuned to perform conservative model selection. Our findings complement those of Knight and Fu (2000) and Fan and Li (2001). We show that the distributions are typically highly nonnormal regardless of how the estimator is tuned, and that this property persists in large samples. The uniform convergence rate of these estimators is also obtained, and is shown to be slower than 1/root(n) in case the estimator is tuned to perform consistent model selection. An impossibility result regarding estimation of the estimators' distribution function is also provided.

References in corpus (13)

• Rejoinder to "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Sparse Estimators and the Oracle Property, or the Return of Hodges' Estimator
• Can one estimate the conditional distribution of post-model-selection estimators?
• The distribution of model averaging estimators and an impossibility result regarding its estimation

Cited by in corpus (27)

• On asymptotically optimal confidence regions and tests for high-dimensional models
• Valid post-selection inference
• Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations
• Estimating Psychological Networks and their Accuracy: A Tutorial Paper
• Double/Debiased Machine Learning for Treatment and Causal Parameters
• On Various Confidence Intervals Post-Model-Selection
• Confidence Sets Based on Penalized Maximum Likelihood Estimators in Gaussian Regression
• Estimating heterogeneous graphical models for discrete data with an application to roll call voting
• Inference in High Dimensions with the Penalized Score Test
• Post-Selection Inference for Generalized Linear Models with Many Controls
• On the Distribution, Model Selection Properties and Uniqueness of the Lasso Estimator in Low and High Dimensions
• Uniformly Valid Post-Regularization Confidence Regions for Many Functional Parameters in Z-Estimation Framework
• Uniformly Valid Confidence Sets Based on the Lasso
• Oracle Inequalities for High-Dimensional Panel Data Models
• Distributional Results for Thresholding Estimators in High-Dimensional Gaussian Regression Models
• Confidence Sets Based on Thresholding Estimators in High-Dimensional Gaussian Regression Models
• Inference for high-dimensional instrumental variables regression
• Empirical priors for prediction in sparse high-dimensional linear regression
• The Loss Rank Criterion for Variable Selection in Linear Regression Analysis
• On Hodges' Superefficiency and Merits of Oracle Property in Model Selection
• Many Average Partial Effects: with An Application to Text Regression
• Approximating posteriors with high-dimensional nuisance parameters via integrated rotated Gaussian approximation
• Uniformly Valid Inference Based on the Lasso in Linear Mixed Models
• Moment convergence in regularized estimation under multiple and mixed-rates asymptotics
• Discussion on "Model Confidence Bounds for Variable Selection" by Yang Li, Yuetian Luo, Davide Ferrari, Xiaonan Hu, and Yichen Qin
• Logistic regression and Ising networks: prediction and estimation when violating lasso assumptions
• Support Recovery with Stochastic Gates: Theory and Application for Linear Models