Gibbs Sampling the Posterior of Neural Networks

Abstract

In this paper, we study sampling from a posterior derived from a neural network. We propose a new probabilistic model consisting of adding noise at every pre- and post-activation in the network, arguing that the resulting posterior can be sampled using an efficient Gibbs sampler. For small models, the Gibbs sampler attains similar performances as the state-of-the-art Markov chain Monte Carlo (MCMC) methods, such as the Hamiltonian Monte Carlo (HMC) or the Metropolis adjusted Langevin algorithm (MALA), both on real and synthetic data. By framing our analysis in the teacher-student setting, we introduce a thermalization criterion that allows us to detect when an algorithm, when run on data with synthetic labels, fails to sample from the posterior. The criterion is based on the fact that in the teacher-student setting we can initialize an algorithm directly at equilibrium.

References in corpus (16)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Stochastic Gradient Hamiltonian Monte Carlo
• Statistical physics of inference: Thresholds and algorithms
• TensorFlow Distributions
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• A Complete Recipe for Stochastic Gradient MCMC
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• What Are Bayesian Neural Network Posteriors Really Like?
• Measuring Sample Quality with Kernels
• Techniques for Learning Binary Stochastic Feedforward Neural Networks
• Natural-Parameter Networks: A Class of Probabilistic Neural Networks
• A Theoretical Framework for Inference Learning
• Do Bayesian Neural Networks Need To Be Fully Stochastic?
• Gibbs Sampling the Posterior of Neural Networks

Cited by in corpus (1)

• Gibbs Sampling the Posterior of Neural Networks