Bayesian neural network unit priors and generalized Weibull-tail property

Abstract

The connection between Bayesian neural networks and Gaussian processes gained a lot of attention in the last few years. Hidden units are proven to follow a Gaussian process limit when the layer width tends to infinity. Recent work has suggested that finite Bayesian neural networks may outperform their infinite counterparts because they adapt their internal representations flexibly. To establish solid ground for future research on finite-width neural networks, our goal is to study the prior induced on hidden units. Our main result is an accurate description of hidden units tails which shows that unit priors become heavier-tailed going deeper, thanks to the introduced notion of generalized Weibull-tail. This finding sheds light on the behavior of hidden units of finite Bayesian neural networks.

arXiv comment16 pages, 2 figures, ACML 2021

References in corpus (7)

• On the Impact of the Activation Function on Deep Neural Networks Training
• Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression
• Understanding Priors in Bayesian Neural Networks at the Unit Level
• Bayesian Neural Network Priors Revisited
• Exact marginal prior distributions of finite Bayesian neural networks
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Precise characterization of the prior predictive distribution of deep ReLU networks