Automatic structured variational inference

Abstract

Stochastic variational inference offers an attractive option as a default method for differentiable probabilistic programming. However, the performance of the variational approach depends on the choice of an appropriate variational family. Here, we introduce automatic structured variational inference (ASVI), a fully automated method for constructing structured variational families, inspired by the closed-form update in conjugate Bayesian models. These convex-update families incorporate the forward pass of the input probabilistic program and can therefore capture complex statistical dependencies. Convex-update families have the same space and time complexity as the input probabilistic program and are therefore tractable for a very large family of models including both continuous and discrete variables. We validate our automatic variational method on a wide range of low- and high-dimensional inference problems. We find that ASVI provides a clear improvement in performance when compared with other popular approaches such as the mean-field approach and inverse autoregressive flows. We provide an open source implementation of ASVI in TensorFlow Probability.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Variational Inference with Normalizing Flows
• Normalizing Flows: An Introduction and Review of Current Methods
• Density estimation using Real NVP
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro
• Edward: A library for probabilistic modeling, inference, and criticism
• Bayesian Recurrent Neural Networks
• Scalable Gradients for Stochastic Differential Equations
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• Automated Variational Inference in Probabilistic Programming
• Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
• Stochastic Variational Inference for Hidden Markov Models
• Joint Distributions for TensorFlow Probability
• Effect Handling for Composable Program Transformations in Edward2
• The Indian Chefs Process

Cited by in corpus (3)

• Automatic Backward Filtering Forward Guiding for Markov processes and graphical models
• Automatic variational inference with cascading flows
• Neuroscience-inspired perception-action in robotics: applying active inference for state estimation, control and self-perception