Stopping Criterion for Active Learning Based on Error Stability

Abstract

Active learning is a framework for supervised learning to improve the predictive performance by adaptively annotating a small number of samples. To realize efficient active learning, both an acquisition function that determines the next datum and a stopping criterion that determines when to stop learning should be considered. In this study, we propose a stopping criterion based on error stability, which guarantees that the change in generalization error upon adding a new sample is bounded by the annotation cost and can be applied to any Bayesian active learning. We demonstrate that the proposed criterion stops active learning at the appropriate timing for various learning models and real datasets.

References in corpus (7)

• Bayesian Active Learning for Classification and Preference Learning
• Learning Active Learning from Data
• A Primer on PAC-Bayesian Learning
• Simpler PAC-Bayesian Bounds for Hostile Data
• Active Learning: Problem Settings and Recent Developments
• Analysis of Stopping Active Learning based on Stabilizing Predictions
• Stopping Active Learning based on Predicted Change of F Measure for Text Classification

Cited by in corpus (1)

• Hitting the Target: Stopping Active Learning at the Cost-Based Optimum