Predictive Inference Is Free with the Jackknife+-after-Bootstrap
arXiv:2002.09025
Abstract
Ensemble learning is widely used in applications to make predictions in complex decision problems---for example, averaging models fitted to a sequence of samples bootstrapped from the available training data. While such methods offer more accurate, stable, and robust predictions and model estimates, much less is known about how to perform valid, assumption-lean inference on the output of these types of procedures. In this paper, we propose the jackknife+-after-bootstrap (J+aB), a procedure for constructing a predictive interval, which uses only the available bootstrapped samples and their corresponding fitted models, and is therefore "free" in terms of the cost of model fitting. The J+aB offers a predictive coverage guarantee that holds with no assumptions on the distribution of the data, the nature of the fitted model, or the way in which the ensemble of models are aggregated---at worst, the failure rate of the predictive interval is inflated by a factor of 2. Our numerical experiments verify the coverage and accuracy of the resulting predictive intervals on real data.
arXiv comment31 pages, 12 figures, 4 tables. To appear in the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada
References in corpus (3)
Cited by in corpus (6)
- Testing for Outliers with Conformal p-values
- Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting
- Nested conformal prediction and quantile out-of-bag ensemble methods
- A general framework for multi-step ahead adaptive conformal heteroscedastic time series forecasting
- A Two-Sample Conditional Distribution Test Using Conformal Prediction and Weighted Rank Sum
- Root-finding Approaches for Computing Conformal Prediction Set