Repulsive Deep Ensembles are Bayesian

Abstract

Deep ensembles have recently gained popularity in the deep learning community for their conceptual simplicity and efficiency. However, maintaining functional diversity between ensemble members that are independently trained with gradient descent is challenging. This can lead to pathologies when adding more ensemble members, such as a saturation of the ensemble performance, which converges to the performance of a single model. Moreover, this does not only affect the quality of its predictions, but even more so the uncertainty estimates of the ensemble, and thus its performance on out-of-distribution data. We hypothesize that this limitation can be overcome by discouraging different ensemble members from collapsing to the same function. To this end, we introduce a kernelized repulsive term in the update rule of the deep ensembles. We show that this simple modification not only enforces and maintains diversity among the members but, even more importantly, transforms the maximum a posteriori inference into proper Bayesian inference. Namely, we show that the training dynamics of our proposed repulsive ensembles follow a Wasserstein gradient flow of the KL divergence with the true posterior. We study repulsive terms in weight and function space and empirically compare their performance to standard ensembles and Bayesian baselines on synthetic and real-world prediction tasks.

References in corpus (35)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Weight Uncertainty in Neural Networks
• Understanding deep learning requires rethinking generalization
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Deep Neural Networks as Gaussian Processes
• Deep Ensembles: A Loss Landscape Perspective
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Scaling description of generalization with number of parameters in deep learning
• Functional Variational Bayesian Neural Networks
• Stein Variational Gradient Descent as Gradient Flow
• Snapshot Ensembles: Train 1, get M for free
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Stochastic Gradient Descent as Approximate Bayesian Inference
• What Are Bayesian Neural Network Posteriors Really Like?
• Gradient Estimators for Implicit Models
• A Spectral Approach to Gradient Estimation for Implicit Distributions
• Understanding and Accelerating Particle-Based Variational Inference
• On the geometry of Stein variational gradient descent
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Robust Out-of-distribution Detection for Neural Networks
• A Non-Asymptotic Analysis for Stein Variational Gradient Descent
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Bayesian Neural Network Priors Revisited
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• BNNpriors: A library for Bayesian neural network inference with different prior distributions
• On Linear Identifiability of Learned Representations
• Symmetry-invariant optimization in deep networks
• Understanding Variational Inference in Function-Space
• A Bayesian Perspective on Training Speed and Model Selection
• Annealed Stein Variational Gradient Descent
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Posterior Meta-Replay for Continual Learning

Cited by in corpus (8)

• Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
• Bayesian posterior approximation with stochastic ensembles
• Quantum Bayesian Neural Networks
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Pathologies in priors and inference for Bayesian transformers
• Greedy Bayesian Posterior Approximation with Deep Ensembles
• Neural Variational Gradient Descent
• A Bayesian Approach to Invariant Deep Neural Networks