Automatic Differentiation Variational Inference

Abstract

Probabilistic modeling is iterative. A scientist posits a simple model, fits it to her data, refines it according to her analysis, and repeats. However, fitting complex models to large data is a bottleneck in this process. Deriving algorithms for new models can be both mathematically and computationally challenging, which makes it difficult to efficiently cycle through the steps. To this end, we develop automatic differentiation variational inference (ADVI). Using our method, the scientist only provides a probabilistic model and a dataset, nothing else. ADVI automatically derives an efficient variational inference algorithm, freeing the scientist to refine and explore many models. ADVI supports a broad class of models-no conjugacy assumptions are required. We study ADVI across ten different models and apply it to a dataset with millions of observations. ADVI is integrated into Stan, a probabilistic programming system; it is available for immediate use.

References in corpus (11)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• Church: a language for generative models
• Automated Variational Inference in Probabilistic Programming
• Automatic Variational Inference in Stan
• The Stan Math Library: Reverse-Mode Automatic Differentiation in C++
• Stochastic gradient variational Bayes for gamma approximating distributions
• On Using Control Variates with Stochastic Approximation for Variational Bayes and its Connection to Stochastic Linear Regression
• Hierarchical Variational Models
• The Variational Gaussian Process

Cited by in corpus (36)

• Variational Inference: A Review for Statisticians
• Inferring change points in the COVID-19 spreading reveals the effectiveness of interventions
• Using stacking to average Bayesian predictive distributions
• A Review of automatic differentiation and its efficient implementation
• Seismic tomography using variational inference methods
• Flares, Rotation, and Planets of the AU Mic System from TESS Observations
• Mapping stellar surfaces II: An interpretable Gaussian process model for light curves
• Simultaneous Multiwavelength Flare Observations of EV Lacertae
• Variational Bayes survival analysis for unemployment modelling
• The Generalized Reparameterization Gradient
• Variational Inference at Glacier Scale
• Low-Cost Energy Meter Calibration Method for Measurement and Verification
• Efficient differentially private learning improves drug sensitivity prediction
• Approximating Bayes in the 21st Century
• Hierarchical Variational Models
• Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions
• Fast and Accurate Estimation of Non-Nested Binomial Hierarchical Models Using Variational Inference
• Reparameterization Gradients through Acceptance-Rejection Sampling Algorithms
• Fast Threshold Tests for Detecting Discrimination
• Learning battery model parameter dynamics from data with recursive Gaussian process regression
• Fast Variational Inference for Bayesian Factor Analysis in Single and Multi-Study Settings
• Bayesian Adaptive Selection of Basis Functions for Functional Data Representation
• VIP -- Variational Inversion Package with example implementations of Bayesian tomographic imaging
• How Bayesian Should Bayesian Optimisation Be?
• Probabilistic Programming with Programmable Variational Inference
• Extending the blended generalized extreme value distribution
• Variational Inference In Pachinko Allocation Machines
• Variational inference based on a subclass of closed skew normals
• Independent finite approximations for Bayesian nonparametric inference
• The sociospatial factors of death: Analyzing effects of geospatially-distributed variables in a Bayesian mortality model for Hong Kong
• Filtering Variational Objectives
• Discussion of "Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing"
• ADAVI: Automatic Dual Amortized Variational Inference Applied To Pyramidal Bayesian Models
• Exploring Variational Deep Q Networks
• Non-conjugate variational Bayes for pseudo-likelihood mixed effect models
• Adversarial Variational Bayes Methods for Tweedie Compound Poisson Mixed Models