Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach

Abstract

Here we present an expository, general analysis of valid post-selection or post-regularization inference about a low-dimensional target parameter, $\alpha$, in the presence of a very high-dimensional nuisance parameter, $\eta$, which is estimated using modern selection or regularization methods. Our analysis relies on high-level, easy-to-interpret conditions that allow one to clearly see the structures needed for achieving valid post-regularization inference. Simple, readily verifiable sufficient conditions are provided for a class of affine-quadratic models. We focus our discussion on estimation and inference procedures based on using the empirical analog of theoretical equations $$M(\alpha ,\eta )=0$$ which identify $\alpha$. Within this structure, we show that setting up such equations in a manner such that the orthogonality/immunization condition $${\partial }_{\eta }M(\alpha ,\eta )=0$$ at the true parameter values is satisfied, coupled with plausible conditions on the smoothness of $M$ and the quality of the estimator $\hat{\eta }$, guarantees that inference on for the main parameter $\alpha$ based on testing or point estimation methods discussed below will be regular despite selection or regularization biases occurring in estimation of $\eta$. In particular, the estimator of $\alpha$ will often be uniformly consistent at the root-$n$ rate and uniformly asymptotically normal even though estimators $\hat{\eta }$ will generally not be asymptotically linear and regular. The uniformity holds over large classes of models that do not impose highly implausible "beta-min" conditions. We also show that inference can be carried out by inverting tests formed from Neyman's $C(\alpha )$ (orthogonal score) statistics.

47 pages

References in corpus (23)

• Simultaneous analysis of Lasso and Dantzig selector
• On asymptotically optimal confidence regions and tests for high-dimensional models
• Least squares after model selection in high-dimensional sparse models
• Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming
• Valid post-selection inference
• A significance test for the lasso
• Lasso-type recovery of sparse representations for high-dimensional data
• Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations
• Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors
• Sparse Estimators and the Oracle Property, or the Return of Hodges' Estimator
• Optimal Inference After Model Selection
• Uniform Post Selection Inference for LAD Regression and Other Z-estimation problems
• High-dimensional instrumental variables regression and confidence sets
• ${\ell }_0$-penalized maximum likelihood for sparse directed acyclic graphs
• Exact Post Model Selection Inference for Marginal Screening
• Exact Post-Selection Inference for Sequential Regression Procedures
• LASSO Methods for Gaussian Instrumental Variables Models
• Multivariate Normal Approximation by Stein's Method: The Concentration Inequality Approach
• Inference in High Dimensions with the Penalized Score Test
• On Semiparametric Exponential Family Graphical Models
• Post-Selection Inference for Generalized Linear Models with Many Controls
• Adaptive testing for the graphical lasso
• A Likelihood Ratio Framework for High Dimensional Semiparametric Regression

Cited by in corpus (4)

• Linear Hypothesis Testing in Dense High-Dimensional Linear Models
• DoubleML -- An Object-Oriented Implementation of Double Machine Learning in R
• Significance testing in non-sparse high-dimensional linear models
• Confounder selection strategies targeting stable treatment effect estimators