Stopping Active Learning based on Predicted Change of F Measure for Text Classification

Abstract

During active learning, an effective stopping method allows users to limit the number of annotations, which is cost effective. In this paper, a new stopping method called Predicted Change of F Measure will be introduced that attempts to provide the users an estimate of how much performance of the model is changing at each iteration. This stopping method can be applied with any base learner. This method is useful for reducing the data annotation bottleneck encountered when building text classification systems.

arXiv comment8 pages, 12 tables; published in Proceedings of the 2019 IEEE 13th International Conference on Semantic Computing (ICSC), Newport Beach, CA, USA, pages 47-54, January 2019

References in corpus (7)

• Learning Active Learning from Data
• Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation
• Support Vector Machine Active Learning Algorithms with Query-by-Committee versus Closest-to-Hyperplane Selection
• Analysis of Stopping Active Learning based on Stabilizing Predictions
• The Use of Unlabeled Data versus Labeled Data for Stopping Active Learning for Text Classification
• Impact of Batch Size on Stopping Active Learning for Text Classification
• Filtering Tweets for Social Unrest

Cited by in corpus (10)

• Active Learning for Network Traffic Classification: A Technical Study
• Active Learning: Problem Settings and Recent Developments
• Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience
• A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification
• The Use of Unlabeled Data versus Labeled Data for Stopping Active Learning for Text Classification
• Computational catalyst discovery: Active classification through myopic multiscale sampling
• Stopping Criterion for Active Learning Based on Error Stability
• Stopping criterion for active learning based on deterministic generalization bounds
• Improving Probabilistic Models in Text Classification via Active Learning
• Early Forecasting of Text Classification Accuracy and F-Measure with Active Learning