Precise characterization of the prior predictive distribution of deep ReLU networks

Abstract

Recent works on Bayesian neural networks (BNNs) have highlighted the need to better understand the implications of using Gaussian priors in combination with the compositional structure of the network architecture. Similar in spirit to the kind of analysis that has been developed to devise better initialization schemes for neural networks (cf. He- or Xavier initialization), we derive a precise characterization of the prior predictive distribution of finite-width ReLU networks with Gaussian weights. While theoretical results have been obtained for their heavy-tailedness, the full characterization of the prior predictive distribution (i.e. its density, CDF and moments), remained unknown prior to this work. Our analysis, based on the Meijer-G function, allows us to quantify the influence of architectural choices such as the width or depth of the network on the resulting shape of the prior predictive distribution. We also formally connect our results to previous work in the infinite width setting, demonstrating that the moments of the distribution converge to those of a normal log-normal mixture in the infinite depth limit. Finally, our results provide valuable guidance on prior design: for instance, controlling the predictive variance with depth- and width-informed priors on the weights of the network.

References in corpus (14)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A New Formula for the BER of Binary Modulations with Dual-Branch Selection over Generalized-K Composite Fading Channels
• The Principles of Deep Learning Theory
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• The Case for Bayesian Deep Learning
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• Products of Independent Gaussian Random Matrices
• Understanding Priors in Bayesian Neural Networks at the Unit Level
• Infinite attention: NNGP and NTK for deep attention networks
• Bayesian Neural Network Priors Revisited
• Exact marginal prior distributions of finite Bayesian neural networks
• On products of Gaussian random variables
• The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization

Cited by in corpus (3)

• Exact marginal prior distributions of finite Bayesian neural networks
• Bayesian neural network unit priors and generalized Weibull-tail property
• Dependence between Bayesian neural network units