Bayesian Neural Networks: Essentials
arXiv:2106.13594
Abstract
Bayesian neural networks utilize probabilistic layers that capture uncertainty over weights and activations, and are trained using Bayesian inference. Since these probabilistic layers are designed to be drop-in replacement of their deterministic counter parts, Bayesian neural networks provide a direct and natural way to extend conventional deep neural networks to support probabilistic deep learning. However, it is nontrivial to understand, design and train Bayesian neural networks due to their complexities. We discuss the essentials of Bayesian neural networks including duality (deep neural networks, probabilistic models), approximate Bayesian inference, Bayesian priors, Bayesian posteriors, and deep variational learning. We use TensorFlow Probability APIs and code examples for illustration. The main problem with Bayesian neural networks is that the architecture of deep neural networks makes it quite redundant, and costly, to account for uncertainty for a large number of successive layers. Hybrid Bayesian neural networks, which use few probabilistic layers judicially positioned in the networks, provide a practical solution.
References in corpus (8)
- Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
- Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning
- Understanding Priors in Bayesian Neural Networks at the Unit Level
- 'In-Between' Uncertainty in Bayesian Neural Networks
- The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning
- On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
- Probabilistic Deep Learning with Probabilistic Neural Networks and Deep Probabilistic Models
- Deep Ensembles from a Bayesian Perspective