Diversity inducing Information Bottleneck in Model Ensembles

Abstract

Although deep learning models have achieved state-of-the-art performance on a number of vision tasks, generalization over high dimensional multi-modal data, and reliable predictive uncertainty estimation are still active areas of research. Bayesian approaches including Bayesian Neural Nets (BNNs) do not scale well to modern computer vision tasks, as they are difficult to train, and have poor generalization under dataset-shift. This motivates the need for effective ensembles which can generalize and give reliable uncertainty estimates. In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning the stochastic latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. We evaluate our method on benchmark datasets: MNIST, CIFAR100, TinyImageNet and MIT Places 2, and compared to the most competitive baselines show significant improvements in classification accuracy, under a shift in the data distribution and in out-of-distribution detection. Code will be released in this url https://github.com/rvl-lab-utoronto/dibs

AAAI 2021. Samarth Sinha* and Homanga Bharadhwaj* contributed equally to this work

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• The information bottleneck method
• Wide Residual Networks
• Weight Uncertainty in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Deep Ensembles: A Loss Landscape Perspective
• Deep Variational Information Bottleneck
• Noise Contrastive Priors for Functional Uncertainty
• InfoBot: Transfer and Exploration via the Information Bottleneck
• MIM: Mutual Information Machine

Cited by in corpus (9)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles
• On Power Laws in Deep Ensembles
• Diverse Ensembles Improve Calibration
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• StackMix: A complementary Mix algorithm
• Uniform Priors for Data-Efficient Transfer
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Greedy Bayesian Posterior Approximation with Deep Ensembles