Dependence between Bayesian neural network units

Abstract

The connection between Bayesian neural networks and Gaussian processes gained a lot of attention in the last few years, with the flagship result that hidden units converge to a Gaussian process limit when the layers width tends to infinity. Underpinning this result is the fact that hidden units become independent in the infinite-width limit. Our aim is to shed some light on hidden units dependence properties in practical finite-width Bayesian neural networks. In addition to theoretical results, we assess empirically the depth and width impacts on hidden units dependence properties.

References in corpus (8)

• Deep Neural Networks as Gaussian Processes
• Understanding Priors in Bayesian Neural Networks at the Unit Level
• Bayesian Neural Network Priors Revisited
• Stable behaviour of infinitely wide deep neural networks
• Exact marginal prior distributions of finite Bayesian neural networks
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Precise characterization of the prior predictive distribution of deep ReLU networks
• Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks