Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning

Abstract

The effectiveness of active learning largely depends on the sampling efficiency of the acquisition function. Expected Loss Reduction (ELR) focuses on a Bayesian estimate of the reduction in classification error, and more general costs fit in the same framework. We propose Bayesian Estimate of Mean Proper Scores (BEMPS) to estimate the increase in strictly proper scores such as log probability or negative mean square error within this framework. We also prove convergence results for this general class of costs. To facilitate better experimentation with the new acquisition functions, we develop a complementary batch AL algorithm that encourages diversity in the vector of expected changes in scores for unlabeled data. To allow high-performance classifiers, we combine deep ensembles, and dynamic validation set construction on pretrained models, and further speed up the ensemble process with the idea of Monte Carlo Dropout. Extensive experiments on both texts and images show that the use of mean square error and log probability with BEMPS yields robust acquisition functions and well-calibrated classifiers, and consistently outperforms the others tested. The advantages of BEMPS over the others are further supported by a set of qualitative analyses, where we visualise their sampling behaviour using data maps and t-SNE plots.

arXiv comment16 pages, TPAMI. arXiv admin note: text overlap with arXiv:2110.14171

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• Bayesian Active Learning for Classification and Preference Learning
• Theory and Applications of Proper Scoring Rules
• Capsule Networks against Medical Imaging Data Challenges
• Diverse mini-batch Active Learning
• A Survey of Active Learning for Text Classification using Deep Neural Networks
• Active Contrastive Learning of Audio-Visual Video Representations
• Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles
• Investigating the Effectiveness of Representations Based on Pretrained Transformer-based Language Models in Active Learning for Labelling Text Datasets

Cited by in corpus (1)

• Fast Fishing: Approximating BAIT for Efficient and Scalable Deep Active Image Classification