Overpruning in Variational Bayesian Neural Networks

Abstract

The motivations for using variational inference (VI) in neural networks differ significantly from those in latent variable models. This has a counter-intuitive consequence; more expressive variational approximations can provide significantly worse predictions as compared to those with less expressive families. In this work we make two contributions. First, we identify a cause of this performance gap, variational over-pruning. Second, we introduce a theoretically grounded explanation for this phenomenon. Our perspective sheds light on several related published results and provides intuition into the design of effective variational approximations of neural networks.

Presented the Advances in Approximate Bayesian Inference workshop at NIPS 2017

References in corpus (1)

• Practical Bayesian Optimization of Machine Learning Algorithms

Cited by in corpus (8)

• 'In-Between' Uncertainty in Bayesian Neural Networks
• Improving and Understanding Variational Continual Learning
• Generalized Variational Continual Learning
• A survey on Variational Autoencoders from a GreenAI perspective
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data
• Bayesian neural networks and dimensionality reduction
• TyXe: Pyro-based Bayesian neural nets for Pytorch