Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

Abstract

The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cyclical stepsize schedule, where larger steps discover new modes, and smaller steps characterize each mode. We also prove non-asymptotic convergence of our proposed algorithm. Moreover, we provide extensive experimental results, including ImageNet, to demonstrate the scalability and effectiveness of cyclical SG-MCMC in learning complex multimodal distributions, especially for fully Bayesian inference with modern deep neural networks.

arXiv commentPublished at ICLR 2020

References in corpus (24)

• SGDR: Stochastic Gradient Descent with Warm Restarts
• Weight Uncertainty in Neural Networks
• The Loss Surfaces of Multilayer Networks
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Qualitatively characterizing neural network optimization problems
• User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• Austerity in MCMC Land: Cutting the Metropolis-Hastings Budget
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• Bayesian Recurrent Neural Networks
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• The Case for Bayesian Deep Learning
• Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
• A Unified Particle-Optimization Framework for Scalable Bayesian Sampling
• Bayesian GAN
• (Non-) asymptotic properties of Stochastic Gradient Langevin Dynamics
• Parallel Markov Chain Monte Carlo
• Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization
• Understanding MCMC Dynamics as Flows on the Wasserstein Space

Cited by in corpus (53)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Deep Ensembles: A Loss Landscape Perspective
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• What Are Bayesian Neural Network Posteriors Really Like?
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Markov Chain Monte Carlo-Based Machine Unlearning: Unlearning What Needs to be Forgotten
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Calibrating AI Models for Few-Shot Demodulation via Conformal Prediction
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
• Laplace Redux -- Effortless Bayesian Deep Learning
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• DeepFlow: History Matching in the Space of Deep Generative Models
• Stochastic Gradient MCMC with Repulsive Forces
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• BNNpriors: A library for Bayesian neural network inference with different prior distributions
• Bayesian Active Meta-Learning for Few Pilot Demodulation and Equalization
• Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting
• Repulsive Deep Ensembles are Bayesian
• Bayesian Inference for Large Scale Image Classification
• Learning Neural Network Subspaces
• Bayesian Graph Neural Networks for Molecular Property Prediction
• Annealed Stein Variational Gradient Descent
• SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
• Prior choice affects ability of Bayesian neural networks to identify unknowns
• Strength of Minibatch Noise in SGD
• A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions
• A statistical theory of cold posteriors in deep neural networks
• Evaluating Scalable Uncertainty Estimation Methods for DNN-Based Molecular Property Prediction
• Bayesian Inference Forgetting
• AMAGOLD: Amortized Metropolis Adjustment for Efficient Stochastic Gradient MCMC
• Stein Variational Gradient Descent with Multiple Kernel
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Gibbs Sampling the Posterior of Neural Networks
• On Stein Variational Neural Network Ensembles
• Diversity and Generalization in Neural Network Ensembles
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
• Bayesian OOD detection with aleatoric uncertainty and outlier exposure
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts
• Accelerating Convergence of Replica Exchange Stochastic Gradient MCMC via Variance Reduction
• JaxSGMC: Modular stochastic gradient MCMC in JAX
• Being a Bit Frequentist Improves Bayesian Neural Networks
• Stochastic Gradient Annealed Importance Sampling for Efficient Online Marginal Likelihood Estimation
• Sparse MoEs meet Efficient Ensembles
• Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty
• Pathologies in priors and inference for Bayesian transformers
• Efficient and Transferable Adversarial Examples from Bayesian Neural Networks
• Structured Stochastic Gradient MCMC