AutoCP: Automated Pipelines for Accurate Prediction Intervals

Abstract

Successful application of machine learning models to real-world prediction problems, e.g. financial forecasting and personalized medicine, has proved to be challenging, because such settings require limiting and quantifying the uncertainty in the model predictions, i.e. providing valid and accurate prediction intervals. Conformal Prediction is a distribution-free approach to construct valid prediction intervals in finite samples. However, the prediction intervals constructed by Conformal Prediction are often (because of over-fitting, inappropriate measures of nonconformity, or other issues) overly conservative and hence inadequate for the application(s) at hand. This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP). Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate while optimizing the interval length to be accurate and less conservative. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.

References in corpus (19)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Recursive Partitioning for Heterogeneous Causal Effects
• Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Least Ambiguous Set-Valued Classifiers with Bounded Error Levels
• Frequentist coverage of adaptive nonparametric Bayesian credible sets
• Reliable Prediction Intervals with Regression Neural Networks
• Batched Large-scale Bayesian Optimization in High-dimensional Spaces
• Nested conformal prediction and quantile out-of-bag ensemble methods
• Conformalized Quantile Regression
• Adaptive, Distribution-Free Prediction Intervals for Deep Networks
• Conformal Inference of Counterfactuals and Individual Treatment Effects
• Classification with Valid and Adaptive Coverage
• Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis
• Conformal prediction with localization
• Robust Recursive Partitioning for Heterogeneous Treatment Effects with Uncertainty Quantification
• Conformal prediction intervals for the individual treatment effect
• Lifelong Bayesian Optimization