Active Learning with Logged Data

Abstract

We consider active learning with logged data, where labeled examples are drawn conditioned on a predetermined logging policy, and the goal is to learn a classifier on the entire population, not just conditioned on the logging policy. Prior work addresses this problem either when only logged data is available, or purely in a controlled random experimentation setting where the logged data is ignored. In this work, we combine both approaches to provide an algorithm that uses logged data to bootstrap and inform experimentation, thus achieving the best of both worlds. Our work is inspired by a connection between controlled random experimentation and active learning, and modifies existing disagreement-based active learning algorithms to exploit logged data.

arXiv commentICML 2018

Cited by in corpus (6)

• On Statistical Bias In Active Learning: How and When To Fix It
• Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedback
• Active Learning for Decision-Making from Imbalanced Observational Data
• Revisiting Model-Agnostic Private Learning: Faster Rates and Active Learning
• Active Learning for Noisy Data Streams Using Weak and Strong Labelers
• The Label Complexity of Active Learning from Observational Data