A tutorial on conformal prediction

Abstract

Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability $\epsilon$, together with a method that makes a prediction $\hat{y}$ of a label $y$, it produces a set of labels, typically containing $\hat{y}$, that also contains $y$ with probability $1-\epsilon$. Conformal prediction can be applied to any method for producing $\hat{y}$: a nearest-neighbor method, a support-vector machine, ridge regression, etc. Conformal prediction is designed for an on-line setting in which labels are predicted successively, each one being revealed before the next is predicted. The most novel and valuable feature of conformal prediction is that if the successive examples are sampled independently from the same distribution, then the successive predictions will be right $1-\epsilon$ of the time, even though they are based on an accumulating dataset rather than on independent datasets. In addition to the model under which successive examples are sampled independently, other on-line compression models can also use conformal prediction. The widely used Gaussian linear model is one of these. This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples. A more comprehensive treatment of the topic is provided in "Algorithmic Learning in a Random World", by Vladimir Vovk, Alex Gammerman, and Glenn Shafer (Springer, 2005).

58 pages, 9 figures

References in corpus (2)

• Hedging predictions in machine learning
• R-Estimates vs. GMM: A Theoretical Case Study of Validity and Efficiency

Cited by in corpus (123)

• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Conformal Prediction: a Unified Review of Theory and New Challenges
• Least Ambiguous Set-Valued Classifiers with Bounded Error Levels
• High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
• Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks
• A Survey on Deep Learning-Based Monocular Spacecraft Pose Estimation: Current State, Limitations and Prospects
• Verified Uncertainty Calibration
• Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting
• Conformal Prediction Interval Estimations with an Application to Day-Ahead and Intraday Power Markets
• Revisiting the Calibration of Modern Neural Networks
• Well-calibrated Confidence Measures for Multi-label Text Classification with a Large Number of Labels
• Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends
• Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout
• Conformal Prediction Under Covariate Shift
• Deep Weighted Averaging Classifiers
• Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees
• Beyond Disagreement-based Agnostic Active Learning
• Conformal Inference of Counterfactuals and Individual Treatment Effects
• Adaptive Conformal Inference Under Distribution Shift
• Concepts and Applications of Conformal Prediction in Computational Drug Discovery
• Operational Calibration: Debugging Confidence Errors for DNNs in the Field
• Valid model-free spatial prediction
• Continuous-time trading and emergence of randomness
• Achieving Equalized Odds by Resampling Sensitive Attributes
• Conformal prediction with localization
• What is a randomization test?
• DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems
• Postprocessing of point predictions for probabilistic forecasting of day-ahead electricity prices: The benefits of using isotonic distributional regression
• A general framework for multi-step ahead adaptive conformal heteroscedastic time series forecasting
• Validity, consonant plausibility measures, and conformal prediction
• Valid inferential models for prediction in supervised learning problems
• Unsupervised Risk Estimation Using Only Conditional Independence Structure
• CERN: Confidence-Energy Recurrent Network for Group Activity Recognition
• Distribution Free Prediction Bands
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically
• Moment Multicalibration for Uncertainty Estimation
• A critical assessment of conformal prediction methods applied in binary classification settings
• Improved conformalized quantile regression
• Efficient Nonparametric Conformal Prediction Regions
• Guaranteed Dynamic Scheduling of Ultra-Reliable Low-Latency Traffic via Conformal Prediction
• Consistent Estimation for Partition-wise Regression and Classification Models
• Interval Neural Networks: Uncertainty Scores
• Conformalized density- and distance-based anomaly detection in time-series data
• Assurance Monitoring of Cyber-Physical Systems with Machine Learning Components
• The Extended Littlestone's Dimension for Learning with Mistakes and Abstentions
• Soft (Gaussian CDE) regression models and loss functions
• Conformal Predictive Safety Filter for RL Controllers in Dynamic Environments
• Assurance Monitoring of Learning Enabled Cyber-Physical Systems Using Inductive Conformal Prediction based on Distance Learning
• Root-finding Approaches for Computing Conformal Prediction Set
• Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation
• Interpretable Uncertainty Quantification in AI for HEP
• Private Prediction Sets
• SLIFER: Investigating Performance and Robustness of Malware Detection Pipelines
• An Investigation on Machine Learning Predictive Accuracy Improvement and Uncertainty Reduction using VAE-based Data Augmentation
• Beyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors
• Exchangeability, Conformal Prediction, and Rank Tests
• Conformal prediction intervals for the individual treatment effect
• Theoretical characterization of uncertainty in high-dimensional linear classification
• Uncertainty Quantification of Surrogate Models using Conformal Prediction
• Selective conformal inference with false coverage-statement rate control
• From Shallow to Deep Interactions Between Knowledge Representation, Reasoning and Machine Learning (Kay R. Amel group)
• Improving Fairness in Criminal Justice Algorithmic Risk Assessments Using Conformal Prediction Sets
• Combining exchangeable p-values
• Mitigating Uncertainty in Document Classification
• Optimising Event-Driven Spiking Neural Network with Regularisation and Cutoff
• Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach
• Anomalous Edge Detection in Edge Exchangeable Social Network Models
• Another Look at Bandwidth-free Inference: a Sample Splitting Approach
• Methods to Compute Prediction Intervals: A Review and New Results
• Bootstrap prediction intervals with asymptotic conditional validity and unconditional guarantees
• Making learning more transparent using conformalized performance prediction
• Approximate Newton-based statistical inference using only stochastic gradients
• Deep Kernel Survival Analysis and Subject-Specific Survival Time Prediction Intervals
• Online Control of the False Coverage Rate and False Sign Rate
• Marginalizing Corrupted Features
• Conformal Rule-Based Multi-label Classification
• Online Multivalid Learning: Means, Moments, and Prediction Intervals
• Uncertainty in Lung Cancer Stage for Outcome Estimation via Set-Valued Classification
• On Convergence of Nearest Neighbor Classifiers over Feature Transformations
• Dual-Splitting Conformal Prediction for Multi-Step Time Series Forecasting
• Few-shot Conformal Prediction with Auxiliary Tasks
• ML-AQP: Query-Driven Approximate Query Processing based on Machine Learning
• Safety Case Templates for Autonomous Systems
• Understanding the Under-Coverage Bias in Uncertainty Estimation
• Transfer learning for Remaining Useful Life Prediction Based on Consensus Self-Organizing Models
• Aggregating Predictions on Multiple Non-disclosed Datasets using Conformal Prediction
• Less is More: Rejecting Unreliable Reviews for Product Question Answering
• Efficient and minimal length parametric conformal prediction regions
• Efficient Conformal Prediction via Cascaded Inference with Expanded Admission
• A Note on Posterior Probability Estimation for Classifiers
• Combining Deep Learning and Verification for Precise Object Instance Detection
• Coverage-Guaranteed Prediction Sets for Out-of-Distribution Data
• Reliable Part-of-Speech Tagging of Historical Corpora through Set-Valued Prediction
• Conformal Anomaly Detection on Spatio-Temporal Observations with Missing Data
• Model-free Bootstrap and Conformal Prediction in Regression: Conditionality, Conjecture Testing, and Pertinent Prediction Intervals
• Homeostasis phenomenon in predictive inference when using a wrong learning model: a tale of random split of data into training and test sets
• A Pvalue-guided Anomaly Detection Approach Combining Multiple Heterogeneous Log Parser Algorithms on IIoT Systems
• Selective Classification via One-Sided Prediction
• T-SCI: A Two-Stage Conformal Inference Algorithm with Guaranteed Coverage for Cox-MLP
• A Nonconformity Approach to Model Selection for SVMs
• Beyond mirkwood: Enhancing SED Modeling with Conformal Predictions
• Discriminative Learning of Prediction Intervals
• Classification with Reject Option: Distribution-free Error Guarantees via Conformal Prediction
• Universal Prediction Band via Semi-Definite Programming
• How Nonconformity Functions and Difficulty of Datasets Impact the Efficiency of Conformal Classifiers
• Inject Machine Learning into Significance Test for Misspecified Linear Models
• Conformal Predictors for Compound Activity Prediction
• Multiple conditional randomization tests for lagged and spillover treatment effects
• A Conformal Prediction Approach to Explore Functional Data
• Data-driven Reachability using Christoffel Functions and Conformal Prediction
• Non-Parametric Methods Applied to the N-Sample Series Comparison
• Conformal Bayesian Computation
• Consistent Accelerated Inference via Confident Adaptive Transformers
• Bin-Conditional Conformal Prediction of Fatalities from Armed Conflict
• Transductive-Inductive Cluster Approximation Via Multivariate Chebyshev Inequality
• Prediction error quantification through probabilistic scaling -- EXTENDED VERSION
• Distribution-Free Prediction Bands for Multivariate Functional Time Series: an Application to the Italian Gas Market
• Computational Challenges in Non-parametric Prediction of Bradycardia in Preterm Infants
• Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration
• Learning Pareto-Efficient Decisions with Confidence
• libconform v0.1.0: a Python library for conformal prediction
• Smart, Adaptive Energy Optimization for Mobile Web Interactions
• Synthesizing Machine Learning Programs with PAC Guarantees via Statistical Sketching