Active Learning for Cost-Sensitive Classification

Abstract

We design an active learning algorithm for cost-sensitive multiclass classification: problems where different errors have different costs. Our algorithm, COAL, makes predictions by regressing to each label's cost and predicting the smallest. On a new example, it uses a set of regressors that perform well on past data to estimate possible costs for each label. It queries only the labels that could be the best, ignoring the sure losers. We prove COAL can be efficiently implemented for any regression family that admits squared loss optimization; it also enjoys strong guarantees with respect to predictive performance and labeling effort. We empirically compare COAL to passive learning and several active learning baselines, showing significant improvements in labeling effort and test cost on real-world datasets.

Fixed typos in Appendix A

Cited by in corpus (12)

• A Contextual Bandit Bake-off
• On Statistical Bias In Active Learning: How and When To Fix It
• Instance-Dependent Complexity of Contextual Bandits and Reinforcement Learning: A Disagreement-Based Perspective
• A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification
• Deep Active Learning for Remote Sensing Object Detection
• Upper Counterfactual Confidence Bounds: a New Optimism Principle for Contextual Bandits
• COVID-19: Strategies for Allocation of Test Kits
• Bayesian Active Learning by Disagreements: A Geometric Perspective
• Online Sub-Sampling for Reinforcement Learning with General Function Approximation
• Improved Algorithms for Efficient Active Learning Halfspaces with Massart and Tsybakov noise
• The Label Complexity of Active Learning from Observational Data
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation