Greedy Bayesian Posterior Approximation with Deep Ensembles

Abstract

Ensembles of independently trained neural networks are a state-of-the-art approach to estimate predictive uncertainty in Deep Learning, and can be interpreted as an approximation of the posterior distribution via a mixture of delta functions. The training of ensembles relies on non-convexity of the loss landscape and random initialization of their individual members, making the resulting posterior approximation uncontrolled. This paper proposes a novel and principled method to tackle this limitation, minimizing an $f$-divergence between the true posterior and a kernel density estimator (KDE) in a function space. We analyze this objective from a combinatorial point of view, and show that it is submodular with respect to mixture components for any $f$. Subsequently, we consider the problem of greedy ensemble construction. From the marginal gain on the negative $f$-divergence, which quantifies an improvement in posterior approximation yielded by adding a new component into the KDE, we derive a novel diversity term for ensemble methods. The performance of our approach is demonstrated on computer vision out-of-distribution detection benchmarks in a range of architectures trained on multiple datasets. The source code of our method is made publicly available at https://github.com/Oulu-IMEDS/greedy_ensembles_training.

arXiv commentPublished in the Transactions on Machine Learning Research: https://openreview.net/forum?id=P1DuPJzVTN

References in corpus (21)

• Scikit-learn: Machine Learning in Python
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Deep Anomaly Detection with Outlier Exposure
• Deep Ensembles: A Loss Landscape Perspective
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Functional Variational Bayesian Neural Networks
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
• Improving Adversarial Robustness of Ensembles with Diversity Training
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles
• Noise Contrastive Priors for Functional Uncertainty
• Repulsive Deep Ensembles are Bayesian
• Diversity inducing Information Bottleneck in Model Ensembles
• Variational f-divergence Minimization
• Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence