Functional Variational Bayesian Neural Networks

Abstract

Variational Bayesian neural networks (BNNs) perform variational inference over weights, but it is difficult to specify meaningful priors and approximate posteriors in a high-dimensional weight space. We introduce functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund (ELBO) defined directly on stochastic processes, i.e. distributions over functions. We prove that the KL divergence between stochastic processes equals the supremum of marginal KL divergences over all finite sets of inputs. Based on this, we introduce a practical training objective which approximates the functional ELBO using finite measurement sets and the spectral Stein gradient estimator. With fBNNs, we can specify priors entailing rich structures, including Gaussian processes and implicit stochastic processes. Empirically, we find fBNNs extrapolate well using various structured priors, provide reliable uncertainty estimates, and scale to large datasets.

ICLR 2019

Cited by in corpus (22)

• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Understanding Variational Inference in Function-Space
• Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series
• Towards Expressive Priors for Bayesian Neural Networks: Poisson Process Radial Basis Function Networks
• The Gaussian Neural Process
• On Stein Variational Neural Network Ensembles
• Scalable Training of Inference Networks for Gaussian-Process Models
• Training on Test Data with Bayesian Adaptation for Covariate Shift
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Sample-efficient reinforcement learning using deep Gaussian processes
• Structured Weight Priors for Convolutional Neural Networks
• Nonparametric Score Estimators
• Scalable Bayesian neural networks by layer-wise input augmentation
• Accurate and Reliable Forecasting using Stochastic Differential Equations
• Bayesian neural networks and dimensionality reduction
• Variational Bayes Neural Network: Posterior Consistency, Classification Accuracy and Computational Challenges
• Estimation of Riemannian distances between covariance operators and Gaussian processes
• Improving Uncertainty Calibration via Prior Augmented Data
• Self-explaining variational posterior distributions for Gaussian Process models