Active Learning from Imperfect Labelers

Abstract

We study active learning where the labeler can not only return incorrect labels but also abstain from labeling. We consider different noise and abstention conditions of the labeler. We propose an algorithm which utilizes abstention responses, and analyze its statistical consistency and query complexity under fairly natural assumptions on the noise and abstention rate of the labeler. This algorithm is adaptive in a sense that it can automatically request less queries with a more informed or less noisy labeler. We couple our algorithm with lower bounds to show that under some technical conditions, it achieves nearly optimal query complexity.

To appear in NIPS 2016

References in corpus (2)

• Agnostic Active Learning Without Constraints
• Active Learning from Weak and Strong Labelers

Cited by in corpus (8)

• Beyond Ads: Sequential Decision-Making Algorithms in Law and Public Policy
• Model-Centric and Data-Centric Aspects of Active Learning for Deep Neural Networks
• Backward-Compatible Prediction Updates: A Probabilistic Approach
• Exponential Savings in Agnostic Active Learning through Abstention
• Deep Active Learning Approach to Adaptive Beamforming for mmWave Initial Alignment
• Active Learning for Noisy Data Streams Using Weak and Strong Labelers
• Disentanglement based Active Learning
• SoQal: Selective Oracle Questioning in Active Learning