Bayesian Neural Network Priors Revisited

Abstract

Isotropic Gaussian priors are the de facto standard for modern Bayesian neural network inference. However, it is unclear whether these priors accurately reflect our true beliefs about the weight distributions or give optimal performance. To find better priors, we study summary statistics of neural network weights in networks trained using stochastic gradient descent (SGD). We find that convolutional neural network (CNN) and ResNet weights display strong spatial correlations, while fully connected networks (FCNNs) display heavy-tailed weight distributions. We show that building these observations into priors can lead to improved performance on a variety of image classification datasets. Surprisingly, these priors mitigate the cold posterior effect in FCNNs, but slightly increase the cold posterior effect in ResNets.

arXiv commentAccepted at ICLR 2022

References in corpus (30)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Weight Uncertainty in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Deep Neural Networks as Gaussian Processes
• Variational Dropout and the Local Reparameterization Trick
• Variational Dropout Sparsifies Deep Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Practical Deep Learning with Bayesian Principles
• Functional Variational Bayesian Neural Networks
• What Are Bayesian Neural Network Posteriors Really Like?
• Noise Contrastive Priors for Functional Uncertainty
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• The Heavy-Tail Phenomenon in SGD
• Learning Invariances using the Marginal Likelihood
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• Bayesian Inference for Large Scale Image Classification
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• The Deep Weight Prior
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Stable behaviour of infinitely wide deep neural networks
• The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks
• Richer priors for infinitely wide multi-layer perceptrons
• A Scale Mixture Perspective of Multiplicative Noise in Neural Networks
• Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks

Cited by in corpus (21)

• What Are Bayesian Neural Network Posteriors Really Like?
• MGP-AttTCN: An Interpretable Machine Learning Model for the Prediction of Sepsis
• Asymptotics of representation learning in finite Bayesian neural networks
• BNNpriors: A library for Bayesian neural network inference with different prior distributions
• Repulsive Deep Ensembles are Bayesian
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Exact marginal prior distributions of finite Bayesian neural networks
• Dangers of Bayesian Model Averaging under Covariate Shift
• Precise characterization of the prior predictive distribution of deep ReLU networks
• On Stein Variational Neural Network Ensembles
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Bayesian OOD detection with aleatoric uncertainty and outlier exposure
• Quantum Bayesian Neural Networks
• Bayesian neural network unit priors and generalized Weibull-tail property
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• Dependence between Bayesian neural network units
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Neural Variational Gradient Descent
• A Bayesian Approach to Invariant Deep Neural Networks
• Pathologies in priors and inference for Bayesian transformers