Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
arXiv:2110.01052
Abstract
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use the framework to provide new calibration methods for several core machine learning tasks, with detailed worked examples in computer vision and tabular medical data.
arXiv commentCode available at https://github.com/aangelopoulos/ltt
References in corpus (13)
- Least Ambiguous Set-Valued Classifiers with Bounded Error Levels
- On Hoeffding's inequalities
- Testing for Outliers with Conformal p-values
- Testing randomness
- Robust Validation: Confident Predictions Even When Distributions Shift
- Conformal Inference of Counterfactuals and Individual Treatment Effects
- Cautious Deep Learning
- A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
- Conformal prediction with localization
- PAC Confidence Sets for Deep Neural Networks via Calibrated Prediction
- Private Prediction Sets
- Conformal calibrators
- PAC Prediction Sets Under Covariate Shift
Cited by in corpus (4)
- Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design
- Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art
- Theoretical characterization of uncertainty in high-dimensional linear classification
- Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling