Sparse Estimators and the Oracle Property, or the Return of Hodges' Estimator

Abstract

We point out some pitfalls related to the concept of an oracle property as used in Fan and Li (2001, 2002, 2004) which are reminiscent of the well-known pitfalls related to Hodges' estimator. The oracle property is often a consequence of sparsity of an estimator. We show that any estimator satisfying a sparsity property has maximal risk that converges to the supremum of the loss function; in particular, the maximal risk diverges to infinity whenever the loss function is unbounded. For ease of presentation the result is set in the framework of a linear regression model, but generalizes far beyond that setting. In a Monte Carlo study we also assess the extent of the problem in finite samples for the smoothly clipped absolute deviation (SCAD) estimator introduced in Fan and Li (2001). We find that this estimator can perform rather poorly in finite samples and that its worst-case performance relative to maximum likelihood deteriorates with increasing sample size when the estimator is tuned to sparsity.

arXiv comment18 pages, 5 figures

References in corpus (2)

• Nonconcave penalized likelihood with a diverging number of parameters
• Consistent covariate selection and post model selection inference in semiparametric regression

Cited by in corpus (39)

• Valid post-selection inference
• Model Selection Techniques -- An Overview
• Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations
• Uniform Post Selection Inference for LAD Regression and Other Z-estimation problems
• Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach
• Plan-Structured Deep Neural Network Models for Query Performance Prediction
• Consistencies and rates of convergence of jump-penalized least squares estimators
• Endogeneity in high dimensions
• Machine Learning Estimation of Heterogeneous Causal Effects: Empirical Monte Carlo Evidence
• Using Invalid Instruments on Purpose: Focused Moment Selection and Averaging for GMM
• A Practical Scheme and Fast Algorithm to Tune the Lasso With Optimality Guarantees
• Double/Debiased Machine Learning for Treatment and Causal Parameters
• Confidence Sets Based on Penalized Maximum Likelihood Estimators in Gaussian Regression
• The GENIUS Approach to Robust Mendelian Randomization Inference
• Evaluation and selection of models for out-of-sample prediction when the sample size is small relative to the complexity of the data-generating process
• Confidence Sets Based on Sparse Estimators Are Necessarily Large
• Estimating heterogeneous graphical models for discrete data with an application to roll call voting
• Sparsity considerations for dependent observations
• Post-Selection Inference for Generalized Linear Models with Many Controls
• Uniformly Valid Post-Regularization Confidence Regions for Many Functional Parameters in Z-Estimation Framework
• Discussion: "A significance test for the lasso"
• On the Distribution of Penalized Maximum Likelihood Estimators: The LASSO, SCAD, and Thresholding
• The costs and benefits of uniformly valid causal inference with high-dimensional nuisance parameters
• Distributional Results for Thresholding Estimators in High-Dimensional Gaussian Regression Models
• Method of Contraction-Expansion (MOCE) for Simultaneous Inference in Linear Models
• On Hodges' Superefficiency and Merits of Oracle Property in Model Selection
• A Generalized Focused Information Criterion for GMM
• Wild Residual Bootstrap Inference for Penalized Quantile Regression with Heteroscedastic Errors
• On the Distribution of the Adaptive LASSO Estimator
• Panel Data Quantile Regression with Grouped Fixed Effects
• Confidence intervals for intentionally biased estimators
• Posterior Convergence and Model Estimation in Bayesian Change-point Problems
• Adaptive Ridge Approach to Heteroscedastic Regression
• Variable Selection in Causal Inference Using Penalization
• Understanding the population structure correction regression
• Minimum Variance Estimation of a Sparse Vector within the Linear Gaussian Model: An RKHS Approach
• Sparse Estimation From Noisy Observations of an Overdetermined Linear System
• Discussion on "Model Confidence Bounds for Variable Selection" by Yang Li, Yuetian Luo, Davide Ferrari, Xiaonan Hu, and Yichen Qin
• Superconsistency of Tests in High Dimensions