Valid inferential models for prediction in supervised learning problems

Abstract

Prediction, where observed data is used to quantify uncertainty about a future observation, is a fundamental problem in statistics. Prediction sets with coverage probability guarantees are a common solution, but these do not provide probabilistic uncertainty quantification in the sense of assigning beliefs to relevant assertions about the future observable. Alternatively, we recommend the use of a {\em probabilistic predictor}, a data-dependent (imprecise) probability distribution for the to-be-predicted observation given the observed data. It is essential that the probabilistic predictor be reliable or valid, and here we offer a notion of validity and explore its behavioral and statistical implications. In particular, we show that valid probabilistic predictors must be imprecise, that they avoid sure loss, and that they lead to prediction procedures with desirable frequentist error rate control properties. We provide a general construction of a provably valid probabilistic predictor, which has close connections to the powerful conformal prediction machinery, and we illustrate this construction in regression and classification applications.

29 pages, 4 figures, 2 tables. Comments welcome at https://researchers.one/articles/21.12.00002

References in corpus (16)

• A tutorial on conformal prediction
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Inferential models: A framework for prior-free posterior probabilistic inference
• Marginal inferential models: prior-free probabilistic inference on interest parameters
• Conditional inferential models: combining information for prior-free probabilistic inference
• False confidence, non-additive beliefs, and valid statistical inference
• Plausibility functions and exact frequentist inference
• On recursive Bayesian predictive distributions
• bamlss: A Lego Toolbox for Flexible Bayesian Regression (and Beyond)
• On an inferential model construction using generalized associations
• Valid model-free spatial prediction
• Prior-free probabilistic prediction of future observations
• Valid inferential models for prediction in supervised learning problems
• Validity, consonant plausibility measures, and conformal prediction
• Generalized inferential models for censored data
• An imprecise-probabilistic characterization of frequentist statistical inference

Cited by in corpus (6)

• Valid inferential models for prediction in supervised learning problems
• Validity, consonant plausibility measures, and conformal prediction
• Direct and approximately valid probabilistic inference on a class of statistical functionals
• Asymptotic efficiency of inferential models and a possibilistic Bernstein--von Mises theorem
• Possibilistic inferential models: a review
• Computationally efficient variational-like approximations of possibilistic inferential models