Few-shot Conformal Prediction with Auxiliary Tasks
arXiv:2102.08898
Abstract
We develop a novel approach to conformal prediction when the target task has limited data available for training. Conformal prediction identifies a small set of promising output candidates in place of a single prediction, with guarantees that the set contains the correct answer with high probability. When training data is limited, however, the predicted set can easily become unusably large. In this work, we obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm over exchangeable collections of auxiliary tasks. Our conformalization algorithm is simple, fast, and agnostic to the choice of underlying model, learning algorithm, or dataset. We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.
arXiv commentICML camera ready
References in corpus (8)
- Prototypical Networks for Few-shot Learning
- On Calibration of Modern Neural Networks
- A tutorial on conformal prediction
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- To Trust Or Not To Trust A Classifier
- Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
- Few-shot Text Classification with Distributional Signatures
- Adaptive, Distribution-Free Prediction Intervals for Deep Networks