Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation

Abstract

We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) introducing a separate path that decomposes the neural net expectation from its variance. We demonstrate formally that introducing separate latent binary variables to the activations allows representing the neural network likelihood as a chain of linear operations. Performing variational inference on this construction enables a sampling-free computation of the evidence lower bound which is a more effective approximation than the widely applied Monte Carlo sampling and CLT related techniques. We evaluate the model on a range of regression and classification tasks against BNN inference alternatives, showing competitive or improved performance over the current state-of-the-art.

arXiv commentAccepted at UAI 2019

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• YOLOv3: An Incremental Improvement
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Variational Inference with Normalizing Flows
• Weight Uncertainty in Neural Networks
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout and the Local Reparameterization Trick
• Gaussian Processes for Big Data
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference
• Reducing Reparameterization Gradient Variance
• Variational Gaussian Dropout is not Bayesian

Cited by in corpus (3)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Bayesian sparsification for deep neural networks with Bayesian model reduction
• Closed Form Variational Objectives For Bayesian Neural Networks with a Single Hidden Layer