Classification with Valid and Adaptive Coverage
arXiv:2006.02544
Abstract
Conformal inference, cross-validation+, and the jackknife+ are hold-out methods that can be combined with virtually any machine learning algorithm to construct prediction sets with guaranteed marginal coverage. In this paper, we develop specialized versions of these techniques for categorical and unordered response labels that, in addition to providing marginal coverage, are also fully adaptive to complex data distributions, in the sense that they perform favorably in terms of approximate conditional coverage compared to alternative methods. The heart of our contribution is a novel conformity score, which we explicitly demonstrate to be powerful and intuitive for classification problems, but whose underlying principle is potentially far more general. Experiments on synthetic and real data demonstrate the practical value of our theoretical guarantees, as well as the statistical advantages of the proposed methods over the existing alternatives.
arXiv comment10 pages, 3 figures; 13 supplementary pages, 4 supplementary figures, 4 supplementary tables
References in corpus (1)
Cited by in corpus (13)
- Testing for Outliers with Conformal p-values
- Conformal Inference of Counterfactuals and Individual Treatment Effects
- Uncertainty Sets for Image Classifiers using Conformal Prediction
- A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
- Set-valued classification -- overview via a unified framework
- Private Prediction Sets
- Exchangeability, Conformal Prediction, and Rank Tests
- Making learning more transparent using conformalized performance prediction
- Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach
- Distribution-Free Federated Learning with Conformal Predictions
- Conformal Anomaly Detection on Spatio-Temporal Observations with Missing Data
- T-SCI: A Two-Stage Conformal Inference Algorithm with Guaranteed Coverage for Cox-MLP
- AutoCP: Automated Pipelines for Accurate Prediction Intervals