Analysis of Stopping Active Learning based on Stabilizing Predictions

Abstract

Within the natural language processing (NLP) community, active learning has been widely investigated and applied in order to alleviate the annotation bottleneck faced by developers of new NLP systems and technologies. This paper presents the first theoretical analysis of stopping active learning based on stabilizing predictions (SP). The analysis has revealed three elements that are central to the success of the SP method: (1) bounds on Cohen's Kappa agreement between successively trained models impose bounds on differences in F-measure performance of the models; (2) since the stop set does not have to be labeled, it can be made large in practice, helping to guarantee that the results transfer to previously unseen streams of examples at test/application time; and (3) good (low variance) sample estimates of Kappa between successive models can be obtained. Proofs of relationships between the level of Kappa agreement and the difference in performance between consecutive models are presented. Specifically, if the Kappa agreement between two models exceeds a threshold T (where $T>0$), then the difference in F-measure performance between those models is bounded above by $\frac{4(1-T)}{T}$ in all cases. If precision of the positive conjunction of the models is assumed to be $p$, then the bound can be tightened to $\frac{4(1-T)}{(p+1)T}$.

10 pages, 8 tables; appeared in Proceedings of the Seventeenth Conference on Computational Natural Language Learning, August 2013

References in corpus (3)

• Assessing agreement on classification tasks: the kappa statistic
• Committee-Based Sample Selection for Probabilistic Classifiers
• Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation

Cited by in corpus (11)

• Discovery of Self-Assembling $\pi$-Conjugated Peptides by Active Learning-Directed Coarse-Grained Molecular Simulation
• Active Learning: Problem Settings and Recent Developments
• Support Vector Machine Active Learning Algorithms with Query-by-Committee versus Closest-to-Hyperplane Selection
• Stopping Active Learning based on Predicted Change of F Measure for Text Classification
• The Use of Unlabeled Data versus Labeled Data for Stopping Active Learning for Text Classification
• Impact of Batch Size on Stopping Active Learning for Text Classification
• Stopping Criterion for Active Learning Based on Error Stability
• Stopping criterion for active learning based on deterministic generalization bounds
• The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions
• Impact of Stop Sets on Stopping Active Learning for Text Classification
• Early Forecasting of Text Classification Accuracy and F-Measure with Active Learning