Bayesian Neural Network Priors Revisited

Abstract

Isotropic Gaussian priors are the de facto standard for modern Bayesian neural network inference. However, it is unclear whether these priors accurately reflect our true beliefs about the weight distributions or give optimal performance. To find better priors, we study summary statistics of neural network weights in networks trained using stochastic gradient descent (SGD). We find that convolutional neural network (CNN) and ResNet weights display strong spatial correlations, while fully connected networks (FCNNs) display heavy-tailed weight distributions. We show that building these observations into priors can lead to improved performance on a variety of image classification datasets. Surprisingly, these priors mitigate the cold posterior effect in FCNNs, but slightly increase the cold posterior effect in ResNets.

Accepted at ICLR 2022

References in corpus (11)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Functional Variational Bayesian Neural Networks
• What Are Bayesian Neural Network Posteriors Really Like?
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Stable behaviour of infinitely wide deep neural networks
• Richer priors for infinitely wide multi-layer perceptrons
• A Scale Mixture Perspective of Multiplicative Noise in Neural Networks

Cited by in corpus (19)

• What Are Bayesian Neural Network Posteriors Really Like?
• Asymptotics of representation learning in finite Bayesian neural networks
• BNNpriors: A library for Bayesian neural network inference with different prior distributions
• Repulsive Deep Ensembles are Bayesian
• Exact marginal prior distributions of finite Bayesian neural networks
• Dangers of Bayesian Model Averaging under Covariate Shift
• Data augmentation in Bayesian neural networks and the cold posterior effect
• On Stein Variational Neural Network Ensembles
• Precise characterization of the prior predictive distribution of deep ReLU networks
• Quantum Bayesian Neural Networks
• Bayesian OOD detection with aleatoric uncertainty and outlier exposure
• Bayesian neural network unit priors and generalized Weibull-tail property
• Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Dependence between Bayesian neural network units
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Pathologies in priors and inference for Bayesian transformers
• A Bayesian Approach to Invariant Deep Neural Networks
• Neural Variational Gradient Descent