Consistent Estimators for Learning to Defer to an Expert
arXiv:2006.01862
Abstract
Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert's decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.
arXiv commentICML 2020
References in corpus (11)
- Wide Residual Networks
- On Calibration of Modern Neural Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Convolutional Neural Networks for Sentence Classification
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- MixMatch: A Holistic Approach to Semi-Supervised Learning
- To Trust Or Not To Trust A Classifier
- Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer
- Active Learning from Weak and Strong Labelers
- Direct Uncertainty Prediction for Medical Second Opinions
- Machine Learning with Abstention for Automated Liver Disease Diagnosis
Cited by in corpus (8)
- No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
- Selective Classification Can Magnify Disparities Across Groups
- A Bandit Model for Human-Machine Decision Making with Private Information and Opacity
- Differentiable Learning Under Triage
- CheXseen: Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays
- Classification Under Human Assistance
- Human-AI Collaboration with Bandit Feedback
- Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possible