Exact posterior distributions of wide Bayesian neural networks

Abstract

Recent work has shown that the prior over functions induced by a deep Bayesian neural network (BNN) behaves as a Gaussian process (GP) as the width of all layers becomes large. However, many BNN applications are concerned with the BNN function space posterior. While some empirical evidence of the posterior convergence was provided in the original works of Neal (1996) and Matthews et al. (2018), it is limited to small datasets or architectures due to the notorious difficulty of obtaining and verifying exactness of BNN posterior approximations. We provide the missing theoretical proof that the exact BNN posterior converges (weakly) to the one induced by the GP limit of the prior. For empirical validation, we show how to generate exact samples from a finite BNN on a small dataset via rejection sampling.

References in corpus (6)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Infinite attention: NNGP and NTK for deep attention networks
• On the infinite width limit of neural networks with a standard parameterization

Cited by in corpus (5)

• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Asymptotics of representation learning in finite Bayesian neural networks
• Depth induces scale-averaging in overparameterized linear Bayesian neural networks
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data