Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outliers

Abstract

Expanding on MacKay (1992), we argue that conventional model-based methods for active learning - like BALD - have a fundamental shortfall: they fail to directly account for the test-time distribution of the input variables. This can lead to pathologies in the acquisition strategy, as what is maximally informative for model parameters may not be maximally informative for prediction: for example, when the data in the pool set is more dispersed than that of the final prediction task, or when the distribution of pool and test samples differs. To correct this, we revisit an acquisition strategy that is based on maximizing the expected information gained about possible future predictions, referring to this as the Expected Predictive Information Gain (EPIG). As EPIG does not scale well for batch acquisition, we further examine an alternative strategy, a hybrid between BALD and EPIG, which we call the Joint Expected Predictive Information Gain (JEPIG). We consider using both for active learning with Bayesian neural networks on a variety of datasets, examining the behavior under distribution shift in the pool set.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Bayesian Active Learning for Classification and Preference Learning
• Deep Anomaly Detection with Outlier Exposure
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• Understanding Measures of Uncertainty for Adversarial Example Detection
• On Nesting Monte Carlo Estimators
• On Statistical Bias In Active Learning: How and When To Fix It
• Predictive Information Accelerates Learning in RL
• SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios