Probabilistic Meta-Representations Of Neural Networks

Abstract

Existing Bayesian treatments of neural networks are typically characterized by weak prior and approximate posterior distributions according to which all the weights are drawn independently. Here, we consider a richer prior distribution in which units in the network are represented by latent variables, and the weights between units are drawn conditionally on the values of the collection of those variables. This allows rich correlations between related weights, and can be seen as realizing a function prior with a Bayesian complexity regularizer ensuring simple solutions. We illustrate the resulting meta-representations and representations, elucidating the power of this prior.

arXiv commentpresented at UAI 2018 Uncertainty In Deep Learning Workshop (UDL AUG. 2018)

Cited by in corpus (8)

• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• Subspace Inference for Bayesian Deep Learning
• Hypermodels for Exploration
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Applying SVGD to Bayesian Neural Networks for Cyclical Time-Series Prediction and Inference
• Posterior Meta-Replay for Continual Learning
• Closed Form Variational Objectives For Bayesian Neural Networks with a Single Hidden Layer
• TyXe: Pyro-based Bayesian neural nets for Pytorch