Efficient Conformal Prediction via Cascaded Inference with Expanded Admission

Abstract

In this paper, we present a novel approach for conformal prediction (CP), in which we aim to identify a set of promising prediction candidates -- in place of a single prediction. This set is guaranteed to contain a correct answer with high probability, and is well-suited for many open-ended classification tasks. In the standard CP paradigm, the predicted set can often be unusably large and also costly to obtain. This is particularly pervasive in settings where the correct answer is not unique, and the number of total possible answers is high. We first expand the CP correctness criterion to allow for additional, inferred "admissible" answers, which can substantially reduce the size of the predicted set while still providing valid performance guarantees. Second, we amortize costs by conformalizing prediction cascades, in which we aggressively prune implausible labels early on by using progressively stronger classifiers -- again, while still providing valid performance guarantees. We demonstrate the empirical effectiveness of our approach for multiple applications in natural language processing and computational chemistry for drug discovery.

arXiv commentICLR 2021. Revision of "Relaxed Conformal Prediction Cascades for Efficient Inference Over Many Labels"

References in corpus (10)

• Scikit-learn: Machine Learning in Python
• On Calibration of Modern Neural Networks
• A tutorial on conformal prediction
• To Trust Or Not To Trust A Classifier
• Learning Recurrent Span Representations for Extractive Question Answering
• Structured Prediction Cascades
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Adaptive, Distribution-Free Prediction Intervals for Deep Networks
• Cascaded Text Generation with Markov Transformers
• Trimmed Conformal Prediction for High-Dimensional Models