Bayesian Inference for Large Scale Image Classification

Abstract

Bayesian inference promises to ground and improve the performance of deep neural networks. It promises to be robust to overfitting, to simplify the training procedure and the space of hyperparameters, and to provide a calibrated measure of uncertainty that can enhance decision making, agent exploration and prediction fairness. Markov Chain Monte Carlo (MCMC) methods enable Bayesian inference by generating samples from the posterior distribution over model parameters. Despite the theoretical advantages of Bayesian inference and the similarity between MCMC and optimization methods, the performance of sampling methods has so far lagged behind optimization methods for large scale deep learning tasks. We aim to fill this gap and introduce ATMC, an adaptive noise MCMC algorithm that estimates and is able to sample from the posterior of a neural network. ATMC dynamically adjusts the amount of momentum and noise applied to each parameter update in order to compensate for the use of stochastic gradients. We use a ResNet architecture without batch normalization to test ATMC on the Cifar10 benchmark and the large scale ImageNet benchmark and show that, despite the absence of batch normalization, ATMC outperforms a strong optimization baseline in terms of both classification accuracy and test log-likelihood. We show that ATMC is intrinsically robust to overfitting on the training data and that ATMC provides a better calibrated measure of uncertainty compared to the optimization baseline.

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Self-Normalizing Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Variational Dropout and the Local Reparameterization Trick
• Bayesian Dark Knowledge
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Fixup Initialization: Residual Learning Without Normalization
• Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo
• WNGrad: Learn the Learning Rate in Gradient Descent
• MCMC-Based Inference in the Era of Big Data: A Fundamental Analysis of the Convergence Complexity of High-Dimensional Chains
• Relativistic Monte Carlo

Cited by in corpus (11)

• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Bayesian Neural Network Priors Revisited
• DeepGLEAM: A hybrid mechanistic and deep learning model for COVID-19 forecasting
• Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting
• Distance-Based Learning from Errors for Confidence Calibration
• The reproducing Stein kernel approach for post-hoc corrected sampling
• Being Bayesian about Categorical Probability
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning
• Langevin Cooling for Domain Translation