Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks

Abstract

Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to the local geometry of parameter space. A second issue is overfitting, which is typically addressed by early stopping. However, recent work has demonstrated that Bayesian model averaging mitigates this problem. The posterior can be sampled by using Stochastic Gradient Langevin Dynamics (SGLD). However, the rapidly changing curvature renders default SGLD methods inefficient. Here, we propose combining adaptive preconditioners with SGLD. In support of this idea, we give theoretical properties on asymptotic convergence and predictive risk. We also provide empirical results for Logistic Regression, Feedforward Neural Nets, and Convolutional Neural Nets, demonstrating that our preconditioned SGLD method gives state-of-the-art performance on these models.

AAAI 2016

References in corpus (17)

• Sequence to Sequence Learning with Neural Networks
• ADADELTA: An Adaptive Learning Rate Method
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Weight Uncertainty in Neural Networks
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Variational Dropout and the Local Reparameterization Trick
• Bayesian Dark Knowledge
• Austerity in MCMC Land: Cutting the Metropolis-Hastings Budget
• Equilibrated adaptive learning rates for non-convex optimization
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring
• Deep Temporal Sigmoid Belief Networks for Sequence Modeling
• Fast Second-Order Stochastic Backpropagation for Variational Inference
• Consistency and fluctuations for stochastic gradient Langevin dynamics

Cited by in corpus (77)

• Preconditioned Stochastic Gradient Descent
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Stein Variational Gradient Descent With Matrix-Valued Kernels
• Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• The True Cost of Stochastic Gradient Langevin Dynamics
• Stochastic Quasi-Newton Langevin Monte Carlo
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Stochastic Gradient MCMC with Repulsive Forces
• Riemannian Stein Variational Gradient Descent for Bayesian Inference
• Meta-Learning for Stochastic Gradient MCMC
• Bayesian Inference for Large Scale Image Classification
• Finding Mixed Nash Equilibria of Generative Adversarial Networks
• Weighted $L^2$-contractivity of Langevin dynamics with singular potentials
• Robust Reinforcement Learning via Adversarial training with Langevin Dynamics
• Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training
• DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module
• Bayesian Graph Convolutional Neural Networks Using Non-Parametric Graph Learning
• Icebreaker: Element-wise Active Information Acquisition with Bayesian Deep Latent Gaussian Model
• EVA: Generating Longitudinal Electronic Health Records Using Conditional Variational Autoencoders
• On the Ergodicity, Bias and Asymptotic Normality of Randomized Midpoint Sampling Method
• A Bayesian Perspective on the Deep Image Prior
• Sampling-based Bayesian Inference with gradient uncertainty
• Bayesian Sparse learning with preconditioned stochastic gradient MCMC and its applications
• A Convergence Analysis for A Class of Practical Variance-Reduction Stochastic Gradient MCMC
• Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server
• Scalable Natural Gradient Langevin Dynamics in Practice
• A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions
• On the Generalised Langevin Equation for Simulated Annealing
• Generalized Bayesian Likelihood-Free Inference
• AMAGOLD: Amortized Metropolis Adjustment for Efficient Stochastic Gradient MCMC
• Non-Parametric Graph Learning for Bayesian Graph Neural Networks
• Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
• Multi-variance replica exchange stochastic gradient MCMC for inverse and forward Bayesian physics-informed neural network
• Data clustering based on Langevin annealing with a self-consistent potential
• Differential Bayesian Neural Nets
• Non-Convex Optimization via Non-Reversible Stochastic Gradient Langevin Dynamics
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
• An adaptive Hessian approximated stochastic gradient MCMC method
• Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory
• DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving
• Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts
• Mini-batch Metropolis-Hastings MCMC with Reversible SGLD Proposal
• Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamics
• Adaptively Preconditioned Stochastic Gradient Langevin Dynamics
• Accelerating Convergence of Replica Exchange Stochastic Gradient MCMC via Variance Reduction
• Stochastic natural gradient descent draws posterior samples in function space
• On Connecting Stochastic Gradient MCMC and Differential Privacy
• Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Sampling with Mirrored Stein Operators
• Probabilistic Approximate Logic and its Implementation in the Logical Imagination Engine
• Deep Poisson gamma dynamical systems
• High-Order Stochastic Gradient Thermostats for Bayesian Learning of Deep Models
• A fast asynchronous MCMC sampler for sparse Bayesian inference
• Stochastic Gradient MCMC with Stale Gradients
• Secure and Differentially Private Bayesian Learning on Distributed Data
• Fast Sampling for Bayesian Max-Margin Models
• Isotropic SGD: a Practical Approach to Bayesian Posterior Sampling
• On Transformations in Stochastic Gradient MCMC
• Variationally Inferred Sampling Through a Refined Bound for Probabilistic Programs
• A Langevinized Ensemble Kalman Filter for Large-Scale Static and Dynamic Learning
• Thompson Sampling via Local Uncertainty
• An Adaptive Empirical Bayesian Method for Sparse Deep Learning
• A Divergence Bound for Hybrids of MCMC and Variational Inference and an Application to Langevin Dynamics and SGVI
• MCMC-Interactive Variational Inference
• Efficient and Transferable Adversarial Examples from Bayesian Neural Networks
• Structured Stochastic Gradient MCMC
• Approximate Inference via Clustering
• Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning