Understanding Priors in Bayesian Neural Networks at the Unit Level

Abstract

We investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, "weight decay", regularization. Our results characterize a more intricate regularization effect at the level of the unit activations. Our main result establishes that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer. We show that first layer units are Gaussian, second layer units are sub-exponential, and units in deeper layers are characterized by sub-Weibull distributions. Our results provide new theoretical insight on deep Bayesian neural networks, which we corroborate with simulation experiments.

arXiv comment10 pages, 5 figures, ICML'19 conference

Cited by in corpus (15)

• Sub-Weibull distributions: generalizing sub-Gaussian and sub-Exponential properties to heavier-tailed distributions
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Exact marginal prior distributions of finite Bayesian neural networks
• Fiedler Regularization: Learning Neural Networks with Graph Sparsity
• Bayesian Neural Networks: Essentials
• Precise characterization of the prior predictive distribution of deep ReLU networks
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Misclassification bounds for PAC-Bayesian sparse deep learning
• A Stochastic Operator Framework for Optimization and Learning with Sub-Weibull Errors
• Bayesian neural network unit priors and generalized Weibull-tail property
• Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• Dependence between Bayesian neural network units
• Bayesian neural networks and dimensionality reduction
• Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning