TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications

Abstract

With the advent of GPU-assisted hardware and maturing high-efficiency software platforms such as TensorFlow and PyTorch, Bayesian posterior sampling for neural networks becomes plausible. In this article we discuss Bayesian parametrization in machine learning based on Markov Chain Monte Carlo methods, specifically discretized stochastic differential equations such as Langevin dynamics and extended system methods in which an ensemble of walkers is employed to enhance sampling. We provide a glimpse of the potential of the sampling-intensive approach by studying (and visualizing) the loss landscape of a neural network applied to the MNIST data set. Moreover, we investigate how the sampling efficiency itself can be significantly enhanced through an ensemble quasi-Newton preconditioning method. This article accompanies the release of a new TensorFlow software package, the Thermodynamic Analytics ToolkIt, which is used in the computational experiments.

arXiv comment25 pages: textual improvements with results unchanged, sections on TATi architecture and software performance removed for size constraints, extended EQN parts, added MNIST nonlinear perceptron example

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Simulated Tempering: A New Monte Carlo Scheme
• Recent Trends in Deep Learning Based Natural Language Processing
• The Loss Surfaces of Multilayer Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Qualitatively characterizing neural network optimization problems
• Essentially No Barriers in Neural Network Energy Landscape
• Sharp Minima Can Generalize For Deep Nets
• Barzilai-Borwein Step Size for Stochastic Gradient Descent
• The computation of averages from equilibrium and nonequilibrium Langevin molecular dynamics
• Perspective: Energy Landscapes for Machine Learning
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• On the Computational Efficiency of Training Neural Networks
• Simulated Tempering Method in the Infinite Switch Limit with Adaptive Weight Learning
• Covariance-Controlled Adaptive Langevin Thermostat for Large-Scale Bayesian Sampling
• Langevin Markov Chain Monte Carlo with stochastic gradients