Conditional predictive inference post model selection

Abstract

We give a finite-sample analysis of predictive inference procedures after model selection in regression with random design. The analysis is focused on a statistically challenging scenario where the number of potentially important explanatory variables can be infinite, where no regularity conditions are imposed on unknown parameters, where the number of explanatory variables in a "good" model can be of the same order as sample size and where the number of candidate models can be of larger order than sample size. The performance of inference procedures is evaluated conditional on the training sample. Under weak conditions on only the number of candidate models and on their complexity, and uniformly over all data-generating processes under consideration, we show that a certain prediction interval is approximately valid and short with high probability in finite samples, in the sense that its actual coverage probability is close to the nominal one and in the sense that its length is close to the length of an infeasible interval that is constructed by actually knowing the "best" candidate model. Similar results are shown to hold for predictive inference procedures other than prediction intervals like, for example, tests of whether a future response will lie above or below a given threshold.

Published in at http://dx.doi.org/10.1214/08-AOS660 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (8)

• Can one estimate the conditional distribution of post-model-selection estimators?
• An adaptation theory for nonparametric confidence intervals
• Adaptive nonparametric confidence sets
• Adaptive confidence balls
• Confidence sets for nonparametric wavelet regression
• Evaluation and selection of models for out-of-sample prediction when the sample size is small relative to the complexity of the data-generating process
• Confidence balls in Gaussian regression
• The distribution of a linear predictor after model selection: Unconditional finite-sample distributions and asymptotic approximations

Cited by in corpus (9)

• Ridge regression and asymptotic minimax estimation over spheres of growing dimension
• A scalable estimate of the extra-sample prediction error via approximate leave-one-out
• The relative effects of dimensionality and multiplicity of hypotheses on the F-test in linear regression
• Empirical priors for prediction in sparse high-dimensional linear regression
• Optimal Estimation and Prediction for Dense Signals in High-Dimensional Linear Models
• Forward-Selected Panel Data Approach for Program Evaluation
• Admissibility of the usual confidence set for the mean of a univariate or bivariate normal population: The unknown-variance case
• Prediction out-of-sample using block shrinkage estimators: model selection and predictive inference
• Discussion on "Model Confidence Bounds for Variable Selection" by Yang Li, Yuetian Luo, Davide Ferrari, Xiaonan Hu, and Yichen Qin