Bayesian Deep Ensembles via the Neural Tangent Kernel

Abstract

We explore the link between deep ensembles and Gaussian processes (GPs) through the lens of the Neural Tangent Kernel (NTK): a recent development in understanding the training dynamics of wide neural networks (NNs). Previous work has shown that even in the infinite width limit, when NNs become GPs, there is no GP posterior interpretation to a deep ensemble trained with squared error loss. We introduce a simple modification to standard deep ensembles training, through addition of a computationally-tractable, randomised and untrainable function to each ensemble member, that enables a posterior interpretation in the infinite width limit. When ensembled together, our trained NNs give an approximation to a posterior predictive distribution, and we prove that our Bayesian deep ensembles make more conservative predictions than standard deep ensembles in the infinite width limit. Finally, using finite width NNs we demonstrate that our Bayesian deep ensembles faithfully emulate the analytic posterior predictive when available, and can outperform standard deep ensembles in various out-of-distribution settings, for both regression and classification tasks.

References in corpus (31)

• Adam: A Method for Stochastic Optimization
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Deep Neural Networks as Gaussian Processes
• Gaussian Processes for Big Data
• On Exact Computation with an Infinitely Wide Neural Net
• Deep Ensembles: A Loss Landscape Perspective
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• On Lazy Training in Differentiable Programming
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Functional Variational Bayesian Neural Networks
• Kernel and Rich Regimes in Overparametrized Models
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Asymptotics of Wide Networks from Feynman Diagrams
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Hydra: Preserving Ensemble Diversity for Model Distillation
• Neural Kernels Without Tangents
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
• On the infinite width limit of neural networks with a standard parameterization
• Approximate Inference Turns Deep Networks into Gaussian Processes

Cited by in corpus (11)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Reliable Neural Networks for Regression Uncertainty Estimation
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• On Power Laws in Deep Ensembles
• Repulsive Deep Ensembles are Bayesian
• A Bayesian Perspective on Training Speed and Model Selection
• Deep Ensembles from a Bayesian Perspective
• Conditional Variational Autoencoder for Learned Image Reconstruction
• Uncertainty-aware Remaining Useful Life predictor
• Quantifying Epistemic Uncertainty in Deep Learning
• Quasi-Bayesian Dual Instrumental Variable Regression