A Survey of Active Learning for Text Classification using Deep Neural Networks

Abstract

Natural language processing (NLP) and neural networks (NNs) have both undergone significant changes in recent years. For active learning (AL) purposes, NNs are, however, less commonly used -- despite their current popularity. By using the superior text classification performance of NNs for AL, we can either increase a model's performance using the same amount of data or reduce the data and therefore the required annotation efforts while keeping the same performance. We review AL for text classification using deep neural networks (DNNs) and elaborate on two main causes which used to hinder the adoption: (a) the inability of NNs to provide reliable uncertainty estimates, on which the most commonly used query strategies rely, and (b) the challenge of training DNNs on small data. To investigate the former, we construct a taxonomy of query strategies, which distinguishes between data-based, model-based, and prediction-based instance selection, and investigate the prevalence of these classes in recent research. Moreover, we review recent NN-based advances in NLP like word embeddings or language models in the context of (D)NNs, survey the current state-of-the-art at the intersection of AL, text classification, and DNNs and relate recent advances in NLP to AL. Finally, we analyze recent work in AL for text classification, connect the respective query strategies to the taxonomy, and outline commonalities and shortcomings. As a result, we highlight gaps in current research and present open research questions.

References in corpus (17)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Distributed Representations of Sentences and Documents
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Weight Uncertainty in Neural Networks
• Bayesian Active Learning for Classification and Preference Learning
• Regularizing and Optimizing LSTM Language Models
• Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
• Adversarial Active Learning for Deep Networks: a Margin Based Approach
• Learning Active Learning from Data
• Generative Adversarial Active Learning
• Discriminative Active Learning
• How to Fine-Tune BERT for Text Classification?
• Learning Algorithms for Active Learning
• Deep Active Learning for Named Entity Recognition
• Support Vector Machine Active Learning Algorithms with Query-by-Committee versus Closest-to-Hyperplane Selection
• Investigating the Effectiveness of Representations Based on Pretrained Transformer-based Language Models in Active Learning for Labelling Text Datasets

Cited by in corpus (5)

• Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization
• Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning
• Active learning based generative design for the discovery of wide bandgap materials
• Impact of Stop Sets on Stopping Active Learning for Text Classification
• ALE: A Simulation-Based Active Learning Evaluation Framework for the Parameter-Driven Comparison of Query Strategies for NLP