Variational Inference using Implicit Distributions

Abstract

Generative adversarial networks (GANs) have given us a great tool to fit implicit generative models to data. Implicit distributions are ones we can sample from easily, and take derivatives of samples with respect to model parameters. These models are highly expressive and we argue they can prove just as useful for variational inference (VI) as they are for generative modelling. Several papers have proposed GAN-like algorithms for inference, however, connections to the theory of VI are not always well understood. This paper provides a unifying review of existing algorithms establishing connections between variational autoencoders, adversarially learned inference, operator VI, GAN-based image reconstruction, and more. Secondly, the paper provides a framework for building new algorithms: depending on the way the variational bound is expressed we introduce prior-contrastive and joint-contrastive methods, and show practical inference algorithms based on either density ratio estimation or denoising.

References in corpus (5)

• Expectation Propagation for approximate Bayesian inference
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Learning in Implicit Generative Models
• Generative Adversarial Nets from a Density Ratio Estimation Perspective
• Adversarial Message Passing For Graphical Models

Cited by in corpus (9)

• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• On denoising autoencoders trained to minimise binary cross-entropy
• Learning Disentangled Representations with Reference-Based Variational Autoencoders
• Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions
• Stochastic Maximum Likelihood Optimization via Hypernetworks
• Adversarially Approximated Autoencoder for Image Generation and Manipulation
• Importance Weighted Hierarchical Variational Inference
• Variational Inference over Non-differentiable Cardiac Simulators using Bayesian Optimization