Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights

Abstract

Probabilistic neural networks are typically modeled with independent weight priors, which do not capture weight correlations in the prior and do not provide a parsimonious interface to express properties in function space. A desirable class of priors would represent weights compactly, capture correlations between weights, facilitate calibrated reasoning about uncertainty, and allow inclusion of prior knowledge about the function space such as periodicity or dependence on contexts such as inputs. To this end, this paper introduces two innovations: (i) a Gaussian process-based hierarchical model for network weights based on unit embeddings that can flexibly encode correlated weight structures, and (ii) input-dependent versions of these weight priors that can provide convenient ways to regularize the function space through the use of kernels defined on contextual inputs. We show these models provide desirable test-time uncertainty estimates on out-of-distribution data, demonstrate cases of modeling inductive biases for neural networks with kernels which help both interpolation and extrapolation from training data, and demonstrate competitive predictive performance on an active learning benchmark.

12 pages main paper, 13 pages appendix

References in corpus (13)

• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Bayesian Active Learning for Classification and Preference Learning
• Variational Dropout and the Local Reparameterization Trick
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• Functional Variational Bayesian Neural Networks
• Reducing Reparameterization Gradient Variance
• The Functional Neural Process
• Noisy Natural Gradient as Variational Inference
• Overpruning in Variational Bayesian Neural Networks
• Deep Kernel Learning
• Probabilistic Meta-Representations Of Neural Networks

Cited by in corpus (2)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes