Functional Variational Bayesian Neural Networks

Abstract

Variational Bayesian neural networks (BNNs) perform variational inference over weights, but it is difficult to specify meaningful priors and approximate posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions. We prove that the KL divergence between stochastic processes equals the supremum of marginal KL divergences over all finite sets of inputs. Based on this, we introduce a practical training objective which approximates the functional ELBO using finite measurement sets and the spectral Stein gradient estimator. With fBNNs, we can specify priors entailing rich structures, including Gaussian processes and implicit stochastic processes. Empirically, we find fBNNs extrapolate well using various structured priors, provide reliable uncertainty estimates, and scale to large datasets.

ICLR 2019

Cited by in corpus (62)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Identifying Causal-Effect Inference Failure with Uncertainty-Aware Models
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• The Functional Neural Process
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Amortized Variational Inference: A Systematic Review
• Variational Bayesian Approximation of Inverse Problems using Sparse Precision Matrices
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• Bayesian Neural Network Priors Revisited
• Action and Perception as Divergence Minimization
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes
• Repulsive Deep Ensembles are Bayesian
• Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Automated Spectral Kernel Learning
• Bayesian Deep Learning via Subnetwork Inference
• Being Bayesian about Categorical Probability
• Variational Implicit Processes
• Understanding Variational Inference in Function-Space
• Stein variational gradient descent on infinite-dimensional space and applications to statistical inverse problems
• Stationary Activations for Uncertainty Calibration in Deep Learning
• Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes
• Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Incorporating Interpretable Output Constraints in Bayesian Neural Networks
• Stochastic Neural Network with Kronecker Flow
• The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks
• Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series
• Compositional uncertainty in deep Gaussian processes
• Towards Expressive Priors for Bayesian Neural Networks: Poisson Process Radial Basis Function Networks
• The Gaussian Neural Process
• Advancing Audio Emotion and Intent Recognition with Large Pre-Trained Models and Bayesian Inference
• On Stein Variational Neural Network Ensembles
• Sample-efficient reinforcement learning using deep Gaussian processes
• Continuous-Time Model-Based Reinforcement Learning
• Training on Test Data with Bayesian Adaptation for Covariate Shift
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Scalable Training of Inference Networks for Gaussian-Process Models
• Structured Weight Priors for Convolutional Neural Networks
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence
• Scalable Bayesian neural networks by layer-wise input augmentation
• Understanding Uncertainty in Bayesian Deep Learning
• Accurate and Reliable Forecasting using Stochastic Differential Equations
• Nonparametric Score Estimators
• Bayesian neural networks and dimensionality reduction
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Thompson Sampling via Local Uncertainty
• Variational Bayes Neural Network: Posterior Consistency, Classification Accuracy and Computational Challenges
• Self-explaining variational posterior distributions for Gaussian Process models
• Improving Uncertainty Calibration via Prior Augmented Data
• Estimation of Riemannian distances between covariance operators and Gaussian processes
• Greedy Bayesian Posterior Approximation with Deep Ensembles
• Stochastic Bayesian Neural Networks