Radial and Directional Posteriors for Bayesian Neural Networks
arXiv:1902.02603
Abstract
We propose a new variational family for Bayesian neural networks. We decompose the variational posterior into two components, where the radial component captures the strength of each neuron in terms of its magnitude; while the directional component captures the statistical dependencies among the weight parameters. The dependencies learned via the directional density provide better modeling performance compared to the widely-used Gaussian mean-field-type variational family. In addition, the strength of input and output neurons learned via the radial density provides a structured way to compress neural networks. Indeed, experiments show that our variational family improves predictive performance and yields compressed networks simultaneously.
arXiv comment17 pages, 9 figures
References in corpus (16)
- Auto-Encoding Variational Bayes
- Categorical Reparameterization with Gumbel-Softmax
- Weight Uncertainty in Neural Networks
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- Variational Dropout and the Local Reparameterization Trick
- Bayesian Compression for Deep Learning
- Variational Dropout Sparsifies Deep Neural Networks
- REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
- Structured Bayesian Pruning via Log-Normal Multiplicative Noise
- Implicit Reparameterization Gradients
- Deep Gaussian Processes for Regression using Approximate Expectation Propagation
- Hyperspherical Variational Auto-Encoders
- Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
- FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
- Computing the Kullback-Leibler Divergence between two Weibull Distributions
- Computing the Kullback-Leibler Divergence between two Generalized Gamma Distributions
Cited by in corpus (5)
- A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
- Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
- Structured Dropout Variational Inference for Bayesian Neural Networks
- Dirichlet Pruning for Neural Network Compression
- On The Distribution of Penultimate Activations of Classification Networks