Optimal Inference After Model Selection
arXiv:1410.2597
Abstract
To perform inference after model selection, we propose controlling the selective type I error; i.e., the error rate of a test given that it was performed. By doing so, we recover long-run frequency properties among selected hypotheses analogous to those that apply in the classical (non-adaptive) context. Our proposal is closely related to data splitting and has a similar intuitive justification, but is more powerful. Exploiting the classical theory of Lehmann and Scheffé (1955), we derive most powerful unbiased selective tests and confidence intervals for inference in exponential family models after arbitrary selection procedures. For linear regression, we derive new selective z-tests that generalize recent proposals for inference after model selection and improve on their power, and new selective t-tests that do not require knowledge of the error variance.
References in corpus (8)
- Valid post-selection inference
- A significance test for the lasso
- Exact Post Model Selection Inference for Marginal Screening
- Exact Post-Selection Inference for Sequential Regression Procedures
- Selective Sequential Model Selection
- A significance test for forward stepwise model selection
- Selective inference with a randomized response
- Adaptive testing for the graphical lasso
Cited by in corpus (88)
- Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach
- Adaptive Concentration of Regression Trees, with Application to Random Forests
- Heterogeneous Employment Effects of Job Search Programmes: A Machine Learning Approach
- Exact Post-Selection Inference for Sequential Regression Procedures
- STAR: A general interactive framework for FDR control under structural constraints
- Selective Sequential Model Selection
- More powerful post-selection inference, with application to the Lasso
- Counterfactual Prediction with Deep Instrumental Variables Networks
- How much does your data exploration overfit? Controlling bias via information usage
- Selective inference for effect modification via the lasso
- Selective inference in regression models with groups of variables
- Bootstrapping and Sample Splitting For High-Dimensional, Assumption-Free Inference
- Testing the simplifying assumption in high-dimensional vine copulas
- Selective inference with a randomized response
- Why Adaptively Collected Data Have Negative Bias and How to Correct for It
- Selective sampling after solving a convex problem
- More Powerful Conditional Selective Inference for Generalized Lasso by Parametric Programming
- On Online Control of False Discovery Rate
- Statistical Methods for Replicability Assessment
- Selective inference after cross-validation
- Selective inference with unknown variance via the square-root LASSO
- On the Post Selection Inference constant under Restricted Isometry Properties
- Generalized Data Thinning Using Sufficient Statistics
- Computing Valid p-value for Optimal Changepoint by Selective Inference using Dynamic Programming
- MAGIC: a general, powerful and tractable method for selective inference
- Sequential Selection Procedures and False Discovery Rate Control
- Bootstrap inference after using multiple queries for model selection
- MMD Aggregated Two-Sample Test
- AdaPT: An interactive procedure for multiple testing with side information
- Controlling False Discovery Rate Using Gaussian Mirrors
- Optimal post-selection inference for sparse signals: a nonparametric empirical-Bayes approach
- Parametric Programming Approach for More Powerful and General Lasso Selective Inference
- A Minimax Theory for Adaptive Data Analysis
- Tree-Values: selective inference for regression trees
- Large Sample Properties of Partitioning-Based Series Estimators
- The Limits of Post-Selection Generalization
- Confidence Intervals for Selected Parameters
- Learning Kernel Tests Without Data Splitting
- Predictive Modeling, Pattern Recognition, and Spatiotemporal Representations of Plant Growth in Simulated and Controlled Environments: A Comprehensive Review
- Exact Post-Selection Inference for Changepoint Detection and Other Generalized Lasso Problems
- An MCMC-free approach to post-selective inference
- False Discovery Rate Control Under General Dependence By Symmetrized Data Aggregation
- Selective inference for the problem of regions via multiscale bootstrap
- Derandomizing Knockoffs
- Tractable Post-Selection Maximum Likelihood Inference for the Lasso
- Forward-Selected Panel Data Approach for Program Evaluation
- Optimal multiple testing and design in clinical trials
- Asymptotics of selective inference
- A Groupwise Approach for Inferring Heterogeneous Treatment Effects in Causal Inference
- Information, Privacy and Stability in Adaptive Data Analysis
- Inference After Selecting Plausibly Valid Instruments with Application to Mendelian Randomization
- More Powerful and General Selective Inference for Stepwise Feature Selection using the Homotopy Continuation Approach
- Valid and Exact Statistical Inference for Multi-dimensional Multiple Change-Points by Selective Inference
- Valid Post-Detection Inference for Change Points Identified Using Trend Filtering
- Inferring Treatment Effects After Testing Instrument Strength in Linear Models
- Post hoc false positive control for spatially structured hypotheses
- Selective Inference for Change Point Detection in Multi-dimensional Sequences
- Sequential Multiple Testing
- Natural Analysts in Adaptive Data Analysis
- A Selective Approach to Internal Inference
- Whiteout: when do fixed-X knockoffs fail?
- Functional Variable Selection for EMG-based Control of a Robotic Hand Prosthetic
- Multiple testing when many -values are uniformly conservative, with application to testing qualitative interaction in educational interventions
- On the Minimum Attainable Risk in Permutation Invariant Problems
- Time to adjust: Improving replicability in experimental psychology by adjustment for evident selective inference
- Markov Neighborhood Regression for High-Dimensional Inference
- Approximate Post-Selective Inference for Regression with the Group LASSO
- More Powerful Selective Kernel Tests for Feature Selection
- Analysis of error control in large scale two-stage multiple hypothesis testing
- Post model-fitting exploration via a "Next-Door" analysis
- Valid Inference Corrected for Outlier Removal
- The Terminating-Random Experiments Selector: Fast High-Dimensional Variable Selection with False Discovery Rate Control
- Selection-adjusted inference: an application to confidence intervals for cis-eQTL effect sizes
- Computing Valid p-values for Image Segmentation by Selective Inference
- Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?
- Selective Inference and Learning Mixed Graphical Models
- Scalable methods for Bayesian selective inference
- Bayesian Selective Inference: Non-informative Priors
- A general framework for estimation and inference from clusters of features
- Quantifying uncertainty in spikes estimated from calcium imaging data
- Selective Inference Approach for Statistically Sound Predictive Pattern Mining
- Post-selection Inference in Regression Models for Group Testing Data
- The Generic Holdout: Preventing False-Discoveries in Adaptive Data Science
- Debiased and threshold ridge regression for linear model with heteroskedastic and dependent error
- Selective Inference for Latent Block Models
- Improved Multiple Confidence Intervals via Thresholding Informed by Prior Information
- Selective Inference via Marginal Screening for High Dimensional Classification
- Selective inference after likelihood- or test-based model selection in linear models