Active Learning from Weak and Strong Labelers

Abstract

An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses active learning with labels obtained from strong and weak labelers, where in addition to the standard active learning setting, we have an extra weak labeler which may occasionally provide incorrect labels. An example is learning to classify medical images where either expensive labels may be obtained from a physician (oracle or strong labeler), or cheaper but occasionally incorrect labels may be obtained from a medical resident (weak labeler). Our goal is to learn a classifier with low error on data labeled by the oracle, while using the weak labeler to reduce the number of label queries made to this labeler. We provide an active learning algorithm for this setting, establish its statistical consistency, and analyze its label complexity to characterize when it can provide label savings over using the strong labeler alone.

To appear in NIPS 2015

References in corpus (2)

• Agnostic Active Learning Without Constraints
• Beyond Disagreement-based Agnostic Active Learning

Cited by in corpus (11)

• A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• Active Learning from Imperfect Labelers
• Consistent Estimators for Learning to Defer to an Expert
• Discovering General-Purpose Active Learning Strategies
• Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedback
• Noisy Blackbox Optimization with Multi-Fidelity Queries: A Tree Search Approach
• Active Learning for Noisy Data Streams Using Weak and Strong Labelers
• SoQal: Selective Oracle Questioning in Active Learning
• Disentanglement based Active Learning