Classification Under Human Assistance

Abstract

Most supervised learning models are trained for full automation. However, their predictions are sometimes worse than those by human experts on some specific instances. Motivated by this empirical observation, our goal is to design classifiers that are optimized to operate under different automation levels. More specifically, we focus on convex margin-based classifiers and first show that the problem is NP-hard. Then, we further show that, for support vector machines, the corresponding objective function can be expressed as the difference of two functions f = g - c, where g is monotone, non-negative and γ-weakly submodular, and c is non-negative and modular. This representation allows a recently introduced deterministic greedy algorithm, as well as a more efficient randomized variant of the algorithm, to enjoy approximation guarantees at solving the problem. Experiments on synthetic and real-world data from several applications in medical diagnosis illustrate our theoretical findings and demonstrate that, under human assistance, supervised learning models trained to operate under different automation levels can outperform those trained for full automation as well as humans operating alone.

arXiv commentPublished in AAAI 2021

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Cooperative Inverse Reinforcement Learning
• Combating Label Noise in Deep Learning Using Abstention
• Guarantees for Greedy Maximization of Non-submodular Functions with Applications
• Learning Policy Representations in Multiagent Systems
• The Algorithmic Automation Problem: Prediction, Triage, and Human Effort
• Submodular Maximization Beyond Non-negativity: Guarantees, Fast Algorithms, and Applications
• Teaching Inverse Reinforcement Learners via Features and Demonstrations
• Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints
• Consistent Estimators for Learning to Defer to an Expert
• Mathematical Models of Adaptation in Human-Robot Collaboration
• Active Regression via Linear-Sample Sparsification
• Learning Not to Learn in the Presence of Noisy Labels
• Active Regression by Stratification
• Towards Deployment of Robust AI Agents for Human-Machine Partnerships
• Non-submodular Function Maximization subject to a Matroid Constraint, with Applications