Valid post-selection inference

Abstract

It is common practice in statistical data analysis to perform data-driven variable selection and derive statistical inference from the resulting model. Such inference enjoys none of the guarantees that classical statistical theory provides for tests and confidence intervals when the model has been chosen a priori. We propose to produce valid ``post-selection inference'' by reducing the problem to one of simultaneous inference and hence suitably widening conventional confidence and retention intervals. Simultaneity is required for all linear functions that arise as coefficient estimates in all submodels. By purchasing ``simultaneity insurance'' for all possible submodels, the resulting post-selection inference is rendered universally valid under all possible model selection procedures. This inference is therefore generally conservative for particular selection procedures, but it is always less conservative than full Scheffe protection. Importantly it does not depend on the truth of the selected submodel, and hence it produces valid inference even in wrong models. We describe the structure of the simultaneous inference problem and give some asymptotic results.

Published in at http://dx.doi.org/10.1214/12-AOS1077 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (7)

• Matching Methods for Causal Inference: A Review and a Look Forward
• Valid post-selection inference
• Sparse Estimators and the Oracle Property, or the Return of Hodges' Estimator
• Can one estimate the conditional distribution of post-model-selection estimators?
• The distribution of model averaging estimators and an impossibility result regarding its estimation
• On the Distribution of Penalized Maximum Likelihood Estimators: The LASSO, SCAD, and Thresholding
• The distribution of a linear predictor after model selection: Unconditional finite-sample distributions and asymptotic approximations

Cited by in corpus (126)

• Valid post-selection inference
• Controlling the false discovery rate via knockoffs
• Exact post-selection inference, with application to the lasso
• Model Selection Techniques -- An Overview
• Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations
• SLOPE - Adaptive variable selection via convex optimization
• Optimal Inference After Model Selection
• High-Dimensional Inference: Confidence Intervals, $p$-Values and R-Software hdi
• Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach
• Adaptive Concentration of Regression Trees, with Application to Random Forests
• Confidence intervals for high-dimensional inverse covariance estimation
• Regularized Ordinal Regression and the ordinalNet R Package
• Rates of convergence of the Adaptive LASSO estimators to the Oracle distribution and higher order refinements by the bootstrap
• Exact Post Model Selection Inference for Marginal Screening
• Least Squares Approximation for a Distributed System
• Exact Post-Selection Inference for Sequential Regression Procedures
• Post hoc inference via joint family-wise error rate control
• Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing
• On Various Confidence Intervals Post-Model-Selection
• Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection
• Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings
• Discovering and Deciphering Relationships Across Disparate Data Modalities
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Selective inference for effect modification via the lasso
• A Simple Adjustment for Bandwidth Snooping
• Inference in High Dimensions with the Penalized Score Test
• Bootstrapping and Sample Splitting For High-Dimensional, Assumption-Free Inference
• Selective inference with a randomized response
• Confounder selection strategies targeting stable treatment effect estimators
• A General Framework for Robust Testing and Confidence Regions in High-Dimensional Quantile Regression
• Rejoinder: "A significance test for the lasso"
• More Powerful Conditional Selective Inference for Generalized Lasso by Parametric Programming
• Discussion: "A significance test for the lasso"
• On the Post Selection Inference constant under Restricted Isometry Properties
• Two Sample Inference for Populations of Graphical Models with Applications to Functional Connectivity
• Evaluating the Effectiveness of Personalized Medicine with Software
• A Bootstrap Lasso + Partial Ridge Method to Construct Confidence Intervals for Parameters in High-dimensional Sparse Linear Models
• Nonparametric Empirical Bayes Estimation on Heterogeneous Data
• A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results
• Spherical Cap Packing Asymptotics and Rank-Extreme Detection
• Post-selection estimation and testing following aggregated association tests
• Floodgate: inference for model-free variable importance
• MAGIC: a general, powerful and tractable method for selective inference
• Uniformly Valid Confidence Sets Based on the Lasso
• Fast and Interpretable Consensus Clustering via Minipatch Learning
• AdaPT: An interactive procedure for multiple testing with side information
• Discussion: "A significance test for the lasso"
• Controlling False Discovery Rate Using Gaussian Mirrors
• The costs and benefits of uniformly valid causal inference with high-dimensional nuisance parameters
• Confidence Sets Based on Thresholding Estimators in High-Dimensional Gaussian Regression Models
• Orthogonal Machine Learning: Power and Limitations
• Discussion: "A significance test for the lasso"
• Graphical tools for selecting conditional instrumental sets
• Exact Post-Selection Inference for Changepoint Detection and Other Generalized Lasso Problems
• An efficient and robust approach to Mendelian randomization with measured pleiotropic effects in a high-dimensional setting
• Low-Complexity Methods for Estimation After Parameter Selection
• Valid Post-selection Inference in Assumption-lean Linear Regression
• Models as Approximations II: A Model-Free Theory of Parametric Regression
• Confidence intervals centered on bootstrap smoothed estimators
• Asymptotics of selective inference
• Forward-Selected Panel Data Approach for Program Evaluation
• Using Regression Kernels to Forecast A Failure to Appear in Court
• On Hodges' Superefficiency and Merits of Oracle Property in Model Selection
• Optimal multiple testing and design in clinical trials
• Likelihood Ratio Test in Multivariate Linear Regression: from Low to High Dimension
• Proximal MCMC for Bayesian Inference of Constrained and Regularized Estimation
• On Selecting and Conditioning in Multiple Testing and Selective Inference
• Tractable Post-Selection Maximum Likelihood Inference for the Lasso
• Information, Privacy and Stability in Adaptive Data Analysis
• A Groupwise Approach for Inferring Heterogeneous Treatment Effects in Causal Inference
• Beyond subjective and objective in statistics
• Informative Goodness-of-Fit for Multivariate Distributions
• Valid Post-Detection Inference for Change Points Identified Using Trend Filtering
• Non-Bayesian Post-Model-Selection Estimation as Estimation Under Model Misspecification
• Post hoc false positive control for spatially structured hypotheses
• Two-Stage Robust and Sparse Distributed Statistical Inference for Large-Scale Data
• Constructing confidence sets after lasso selection by randomized estimator augmentation
• More Powerful and General Selective Inference for Stepwise Feature Selection using the Homotopy Continuation Approach
• Discovery and inference of a causal network with hidden confounding
• EigenPrism: Inference for High-Dimensional Signal-to-Noise Ratios
• Covariate Selection Based on a Assumpton-free Approach to Linear Regression with Exact Probabilities
• A Lasso-OLS Hybrid Approach to Covariate Selection and Average Treatment Effect Estimation for Clustered RCTs Using Design-Based Methods
• Post-selection inference with HSIC-Lasso
• Uniformly Valid Inference Based on the Lasso in Linear Mixed Models
• Fitting High-Dimensional Interaction Models with Error Control
• Use Of Vapnik-Chervonenkis Dimension in Model Selection
• Inference after black box selection
• Post hoc false discovery proportion inference under a Hidden Markov Model
• Admissibility of the usual confidence set for the mean of a univariate or bivariate normal population: The unknown-variance case
• A Normality Test for High-dimensional Data based on a Nearest Neighbor Approach
• Post-selection inference for linear mixed model parameters using the conditional Akaike information criterion
• Valid uncertainty quantification about the model in a linear regression setting
• Model selection for estimation of causal parameters
• Algorithm-agnostic significance testing in supervised learning with multimodal data
• Unlucky Number 13? Manipulating Evidence Subject to Snooping
• A Multi-resolution Theory for Approximating Infinite-$p$-Zero-$n$: Transitional Inference, Individualized Predictions, and a World Without Bias-Variance Trade-off
• Markov Neighborhood Regression for High-Dimensional Inference
• Improving Power by Conditioning on Less in Post-selection Inference for Changepoints
• Selection-adjusted inference: an application to confidence intervals for cis-eQTL effect sizes
• Post model-fitting exploration via a "Next-Door" analysis
• Prediction out-of-sample using block shrinkage estimators: model selection and predictive inference
• Submodularity in Statistics: Comparing the Success of Model Selection Methods
• Selective Inference in Propensity Score Analysis
• Mathematical modelling, selection and hierarchical inference to determine the minimal dose in IFN$\alpha$ therapy against Myeloproliferative Neoplasms
• Selective inference after convex clustering with ${\ell }_1$ penalization
• Post-Model-Selection Statistical Inference with Interrupted Time Series Designs: An Evaluation of an Assault Weapons Ban in California
• Inference for High Dimensional Censored Quantile Regression
• Statistical Inference in High-dimensional Generalized Linear Models with Streaming Data
• Weak Signal Identification and Inference in Penalized Model Selection
• Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?
• Selective Inference and Learning Mixed Graphical Models
• revisit: a Workflow Tool for Data Science
• On overfitting and post-selection uncertainty assessments
• An Efficient Post-Selection Inference on High-Order Interaction Models
• Selective Inference for Latent Block Models
• DebiNet: Debiasing Linear Models with Nonlinear Overparameterized Neural Networks
• Inject Machine Learning into Significance Test for Misspecified Linear Models
• All of Linear Regression
• Spatial Variable Selection and An Application to Virginia Lyme Disease Emergence
• Confidently Comparing Estimators with the c-value
• Selective Inference Approach for Statistically Sound Predictive Pattern Mining
• Selective inference after likelihood- or test-based model selection in linear models
• A Computationally Efficient Approach to False Discovery Rate Control and Power Maximisation via Randomisation and Mirror Statistic
• Kernel Ordinary Differential Equations
• Bayesian Selective Inference: Non-informative Priors
• Post-selection Inference in Regression Models for Group Testing Data