Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
arXiv:1905.06076
Abstract
A simple, flexible approach to creating expressive priors in Gaussian process (GP) models makes new kernels from a combination of basic kernels, e.g. summing a periodic and linear kernel can capture seasonal variation with a long term trend. Despite a well-studied link between GPs and Bayesian neural networks (BNNs), the BNN analogue of this has not yet been explored. This paper derives BNN architectures mirroring such kernel combinations. Furthermore, it shows how BNNs can produce periodic kernels, which are often useful in this context. These ideas provide a principled approach to designing BNNs that incorporate prior knowledge about a function. We showcase the practical value of these ideas with illustrative experiments in supervised and reinforcement learning settings.
References in corpus (9)
- Weight Uncertainty in Neural Networks
- Searching for Activation Functions
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- Structure Discovery in Nonparametric Regression through Compositional Kernel Search
- Gaussian Process Kernels for Pattern Discovery and Extrapolation
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Functional Variational Bayesian Neural Networks
- Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning
- Bayesian Neural Network Ensembles
Cited by in corpus (12)
- A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
- Uncertainty in Neural Networks: Approximately Bayesian Ensembling
- How Good is the Bayes Posterior in Deep Neural Networks Really?
- Bayesian Neural Network Priors Revisited
- All You Need is a Good Functional Prior for Bayesian Deep Learning
- Bayesian Deep Learning via Subnetwork Inference
- Stationary Activations for Uncertainty Calibration in Deep Learning
- The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks
- Informative Bayesian Neural Network Priors for Weak Signals
- Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning
- Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
- Gaussian Processes with Skewed Laplace Spectral Mixture Kernels for Long-term Forecasting