Hedging predictions in machine learning

Abstract

Recent advances in machine learning make it possible to design efficient prediction algorithms for data sets with huge numbers of parameters. This paper describes a new technique for "hedging" the predictions output by many such algorithms, including support vector machines, kernel ridge regression, kernel nearest neighbours, and by many other state-of-the-art methods. The hedged predictions for the labels of new objects include quantitative measures of their own accuracy and reliability. These measures are provably valid under the assumption of randomness, traditional in machine learning: the objects and their labels are assumed to be generated independently from the same probability distribution. In particular, it becomes possible to control (up to statistical fluctuations) the number of erroneous predictions by selecting a suitable confidence level. Validity being achieved automatically, the remaining goal of hedged prediction is efficiency: taking full account of the new objects' features and other available information to produce as accurate predictions as possible. This can be done successfully using the powerful machinery of modern machine learning.

24 pages; 9 figures; 2 tables; a version of this paper (with discussion and rejoinder) is to appear in "The Computer Journal"

Cited by in corpus (11)

• A tutorial on conformal prediction
• Regression Conformal Prediction with Nearest Neighbours
• Conformal Inference of Counterfactuals and Individual Treatment Effects
• Adaptive Conformal Inference Under Distribution Shift
• SoftED: Metrics for Soft Evaluation of Time Series Event Detection
• Boost AI Power: Data Augmentation Strategies with unlabelled Data and Conformal Prediction, a Case in Alternative Herbal Medicine Discrimination with Electronic Nose
• Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach
• Anomalous Edge Detection in Edge Exchangeable Social Network Models
• Discriminative Learning of Prediction Intervals
• Conformal Predictors for Compound Activity Prediction
• Social Learning and the Accuracy-Risk Trade-off in the Wisdom of the Crowd