Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations

Abstract

We challenge the longstanding assumption that the mean-field approximation for variational inference in Bayesian neural networks is severely restrictive, and show this is not the case in deep networks. We prove several results indicating that deep mean-field variational weight posteriors can induce similar distributions in function-space to those induced by shallower networks with complex weight posteriors. We validate our theoretical contributions empirically, both through examination of the weight posterior using Hamiltonian Monte Carlo in small models and by comparing diagonal- to structured-covariance in large settings. Since complex variational posteriors are often expensive and cumbersome to implement, our results suggest that using mean-field variational inference in a deeper model is both a practical and theoretically justified alternative to structured approximations.

arXiv commentAdvances In Neural Information Processing Systems. 2020

References in corpus (10)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On the Convergence of Adam and Beyond
• Variational Inference with Normalizing Flows
• On the Number of Linear Regions of Deep Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Neural Variational Inference and Learning in Belief Networks
• Functional Variational Bayesian Neural Networks
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance
• Introducing an Explicit Symplectic Integration Scheme for Riemannian Manifold Hamiltonian Monte Carlo

Cited by in corpus (10)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• What Are Bayesian Neural Network Posteriors Really Like?
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Calibrating AI Models for Few-Shot Demodulation via Conformal Prediction
• Laplace Redux -- Effortless Bayesian Deep Learning
• Bayesian Active Meta-Learning for Few Pilot Demodulation and Equalization
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Bayesian Deep Learning via Subnetwork Inference
• Inferring the Langevin Equation with Uncertainty via Bayesian Neural Networks
• Transformation Models for Flexible Posteriors in Variational Bayes