SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios

Abstract

Active learning has proven to be useful for minimizing labeling costs by selecting the most informative samples. However, existing active learning methods do not work well in realistic scenarios such as imbalance or rare classes, out-of-distribution data in the unlabeled set, and redundancy. In this work, we propose SIMILAR (Submodular Information Measures based actIve LeARning), a unified active learning framework using recently proposed submodular information measures (SIM) as acquisition functions. We argue that SIMILAR not only works in standard active learning, but also easily extends to the realistic settings considered above and acts as a one-stop solution for active learning that is scalable to large real-world datasets. Empirically, we show that SIMILAR significantly outperforms existing active learning algorithms by as much as ~5% - 18% in the case of rare classes and ~5% - 10% in the case of out-of-distribution data on several image classification tasks like CIFAR-10, MNIST, and ImageNet. SIMILAR is available as a part of the DISTIL toolkit: "https://github.com/decile-team/distil".

To Appear In Thirty-fifth Conference on Neural Information Processing Systems, NeurIPS 2021

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Character-level Convolutional Networks for Text Classification
• SGDR: Stochastic Gradient Descent with Warm Restarts
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
• Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity
• A Memoization Framework for Scaling Submodular Optimization to Large Scale Problems

Cited by in corpus (5)

• Deep Active Learning for Computer Vision: Past and Future
• Forgetful Active Learning with Switch Events: Efficient Sampling for Out-of-Distribution Data
• Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outliers
• Evolution-aware VAriance (EVA) Coreset Selection for Medical Image Classification
• DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptation