Optimal Inference After Model Selection

Abstract

To perform inference after model selection, we propose controlling the selective type I error; i.e., the error rate of a test given that it was performed. By doing so, we recover long-run frequency properties among selected hypotheses analogous to those that apply in the classical (non-adaptive) context. Our proposal is closely related to data splitting and has a similar intuitive justification, but is more powerful. Exploiting the classical theory of Lehmann and Scheffé (1955), we derive most powerful unbiased selective tests and confidence intervals for inference in exponential family models after arbitrary selection procedures. For linear regression, we derive new selective z-tests that generalize recent proposals for inference after model selection and improve on their power, and new selective t-tests that do not require knowledge of the error variance.

References in corpus (5)

• Exact Post Model Selection Inference for Marginal Screening
• Exact Post-Selection Inference for Sequential Regression Procedures
• Selective Sequential Model Selection
• A significance test for forward stepwise model selection
• Selective inference with a randomized response

Cited by in corpus (87)

• Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach
• Adaptive Concentration of Regression Trees, with Application to Random Forests
• Heterogeneous Employment Effects of Job Search Programmes: A Machine Learning Approach
• Exact Post-Selection Inference for Sequential Regression Procedures
• STAR: A general interactive framework for FDR control under structural constraints
• Selective Sequential Model Selection
• More powerful post-selection inference, with application to the Lasso
• Counterfactual Prediction with Deep Instrumental Variables Networks
• How much does your data exploration overfit? Controlling bias via information usage
• Selective inference for effect modification via the lasso
• Selective inference in regression models with groups of variables
• Selective inference with a randomized response
• Testing the simplifying assumption in high-dimensional vine copulas
• Why Adaptively Collected Data Have Negative Bias and How to Correct for It
• Bootstrapping and Sample Splitting For High-Dimensional, Assumption-Free Inference
• Selective sampling after solving a convex problem
• More Powerful Conditional Selective Inference for Generalized Lasso by Parametric Programming
• On Online Control of False Discovery Rate
• Statistical Methods for Replicability Assessment
• Selective inference after cross-validation
• Selective inference with unknown variance via the square-root LASSO
• On the Post Selection Inference constant under Restricted Isometry Properties
• Generalized Data Thinning Using Sufficient Statistics
• Computing Valid p-value for Optimal Changepoint by Selective Inference using Dynamic Programming
• MAGIC: a general, powerful and tractable method for selective inference
• Bootstrap inference after using multiple queries for model selection
• AdaPT: An interactive procedure for multiple testing with side information
• Optimal post-selection inference for sparse signals: a nonparametric empirical-Bayes approach
• MMD Aggregated Two-Sample Test
• Parametric Programming Approach for More Powerful and General Lasso Selective Inference
• Controlling False Discovery Rate Using Gaussian Mirrors
• Tree-Values: selective inference for regression trees
• A Minimax Theory for Adaptive Data Analysis
• Large Sample Properties of Partitioning-Based Series Estimators
• The Limits of Post-Selection Generalization
• Confidence Intervals for Selected Parameters
• Learning Kernel Tests Without Data Splitting
• An MCMC-free approach to post-selective inference
• Exact Post-Selection Inference for Changepoint Detection and Other Generalized Lasso Problems
• Predictive Modeling, Pattern Recognition, and Spatiotemporal Representations of Plant Growth in Simulated and Controlled Environments: A Comprehensive Review
• Derandomizing Knockoffs
• False Discovery Rate Control Under General Dependence By Symmetrized Data Aggregation
• Selective inference for the problem of regions via multiscale bootstrap
• Asymptotics of selective inference
• Optimal multiple testing and design in clinical trials
• Forward-Selected Panel Data Approach for Program Evaluation
• Tractable Post-Selection Maximum Likelihood Inference for the Lasso
• Valid and Exact Statistical Inference for Multi-dimensional Multiple Change-Points by Selective Inference
• Information, Privacy and Stability in Adaptive Data Analysis
• Inference After Selecting Plausibly Valid Instruments with Application to Mendelian Randomization
• More Powerful and General Selective Inference for Stepwise Feature Selection using the Homotopy Continuation Approach
• Valid Post-Detection Inference for Change Points Identified Using Trend Filtering
• Inferring Treatment Effects After Testing Instrument Strength in Linear Models
• A Groupwise Approach for Inferring Heterogeneous Treatment Effects in Causal Inference
• Post hoc false positive control for spatially structured hypotheses
• Sequential Multiple Testing
• Selective Inference for Change Point Detection in Multi-dimensional Sequences
• A Selective Approach to Internal Inference
• Natural Analysts in Adaptive Data Analysis
• Whiteout: when do fixed-X knockoffs fail?
• Functional Variable Selection for EMG-based Control of a Robotic Hand Prosthetic
• Multiple testing when many $p$-values are uniformly conservative, with application to testing qualitative interaction in educational interventions
• On the Minimum Attainable Risk in Permutation Invariant Problems
• Time to adjust: Improving replicability in experimental psychology by adjustment for evident selective inference
• Markov Neighborhood Regression for High-Dimensional Inference
• Approximate Post-Selective Inference for Regression with the Group LASSO
• Analysis of error control in large scale two-stage multiple hypothesis testing
• Selective Inference and Learning Mixed Graphical Models
• The Terminating-Random Experiments Selector: Fast High-Dimensional Variable Selection with False Discovery Rate Control
• Computing Valid p-values for Image Segmentation by Selective Inference
• More Powerful Selective Kernel Tests for Feature Selection
• Valid Inference Corrected for Outlier Removal
• Selection-adjusted inference: an application to confidence intervals for cis-eQTL effect sizes
• Post model-fitting exploration via a "Next-Door" analysis
• Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?
• Scalable methods for Bayesian selective inference
• Selective Inference via Marginal Screening for High Dimensional Classification
• A general framework for estimation and inference from clusters of features
• Selective Inference for Latent Block Models
• Improved Multiple Confidence Intervals via Thresholding Informed by Prior Information
• Quantifying uncertainty in spikes estimated from calcium imaging data
• Bayesian Selective Inference: Non-informative Priors
• Selective inference after likelihood- or test-based model selection in linear models
• Post-selection Inference in Regression Models for Group Testing Data
• Selective Inference Approach for Statistically Sound Predictive Pattern Mining
• The Generic Holdout: Preventing False-Discoveries in Adaptive Data Science
• Debiased and threshold ridge regression for linear model with heteroskedastic and dependent error