Automatic Variational Inference in Stan

Abstract

Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult to automate. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI). The user only provides a Bayesian model and a dataset; nothing else. We make no conjugacy assumptions and support a broad class of models. The algorithm automatically determines an appropriate variational family and optimizes the variational objective. We implement ADVI in Stan (code available now), a probabilistic programming framework. We compare ADVI to MCMC sampling across hierarchical generalized linear models, nonconjugate matrix factorization, and a mixture model. We train the mixture model on a quarter million images. With ADVI we can use variational inference on any model we write in Stan.

References in corpus (3)

• Adam: A Method for Stochastic Optimization
• Automated Variational Inference in Probabilistic Programming
• On Using Control Variates with Stochastic Approximation for Variational Bayes and its Connection to Stochastic Linear Regression

Cited by in corpus (25)

• Automatic Differentiation Variational Inference
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• Stochastic Gradient Descent as Approximate Bayesian Inference
• Scaling Exact Inference for Discrete Probabilistic Programs
• Denotational validation of higher-order Bayesian inference
• A Variational Analysis of Stochastic Gradient Algorithms
• Boosting Variational Inference
• A Factor Graph Approach to Automated Design of Bayesian Signal Processing Algorithms
• Stochastic gradient variational Bayes for gamma approximating distributions
• Variational Bayes survival analysis for unemployment modelling
• The Generalized Reparameterization Gradient
• Hierarchical Variational Models
• Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
• Overdispersed Black-Box Variational Inference
• Swift: Compiled Inference for Probabilistic Programming Languages
• Bayesian Incremental Learning for Deep Neural Networks
• PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference
• Neural Density Estimation and Likelihood-free Inference
• Automatic Variational ABC
• A Goal-Based Movement Model for Continuous Multi-Agent Tasks
• Posterior Dispersion Indices
• Bayesian model selection in the $\mathcal{M}$-open setting -- Approximate posterior inference and probability-proportional-to-size subsampling for efficient large-scale leave-one-out cross-validation
• Point process models for quasi-periodic volcanic earthquakes
• Capturing Structure Implicitly from Time-Series having Limited Data
• Choice functions based multi-objective Bayesian optimisation