Overpruning in Variational Bayesian Neural Networks

Abstract

The motivations for using variational inference (VI) in neural networks differ significantly from those in latent variable models. This has a counter-intuitive consequence; more expressive variational approximations can provide significantly worse predictions as compared to those with less expressive families. In this work we make two contributions. First, we identify a cause of this performance gap, variational over-pruning. Second, we introduce a theoretically grounded explanation for this phenomenon. Our perspective sheds light on several related published results and provides intuition into the design of effective variational approximations of neural networks.

Presented the Advances in Approximate Bayesian Inference workshop at NIPS 2017

References in corpus (1)

• Practical Bayesian Optimization of Machine Learning Algorithms

Cited by in corpus (15)

• Task Agnostic Continual Learning Using Online Variational Bayes
• Conditional Density Estimation with Bayesian Normalising Flows
• 'In-Between' Uncertainty in Bayesian Neural Networks
• Improving and Understanding Variational Continual Learning
• Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
• Generalized Variational Continual Learning
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Kernel Implicit Variational Inference
• A survey on Variational Autoencoders from a GreenAI perspective
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Doubly Semi-Implicit Variational Inference
• Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data
• Variational Bayesian Dropout with a Hierarchical Prior
• Bayesian neural networks and dimensionality reduction
• TyXe: Pyro-based Bayesian neural nets for Pytorch