Variational Inference using Implicit Distributions

Abstract

Generative adversarial networks (GANs) have given us a great tool to fit implicit generative models to data. Implicit distributions are ones we can sample from easily, and take derivatives of samples with respect to model parameters. These models are highly expressive and we argue they can prove just as useful for variational inference (VI) as they are for generative modelling. Several papers have proposed GAN-like algorithms for inference, however, connections to the theory of VI are not always well understood. This paper provides a unifying review of existing algorithms establishing connections between variational autoencoders, adversarially learned inference, operator VI, GAN-based image reconstruction, and more. Secondly, the paper provides a framework for building new algorithms: depending on the way the variational bound is expressed we introduce prior-contrastive and joint-contrastive methods, and show practical inference algorithms based on either density ratio estimation or denoising.

References in corpus (9)

• Auto-Encoding Variational Bayes
• Expectation Propagation for approximate Bayesian inference
• Improved Techniques for Training GANs
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Adversarial Autoencoders
• Learning in Implicit Generative Models
• Improving Variational Inference with Inverse Autoregressive Flow
• Generative Adversarial Nets from a Density Ratio Estimation Perspective
• Adversarial Message Passing For Graphical Models

Cited by in corpus (48)

• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Boundary-Seeking Generative Adversarial Networks
• Deep Active Inference
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Implicit Weight Uncertainty in Neural Networks
• On denoising autoencoders trained to minimise binary cross-entropy
• Gradient Estimators for Implicit Models
• A Spectral Approach to Gradient Estimation for Implicit Distributions
• Approximate Inference with Amortised MCMC
• Advances in Variational Inference
• Semi-Implicit Variational Inference
• Learning Disentangled Representations with Reference-Based Variational Autoencoders
• Adversarial Variational Optimization of Non-Differentiable Simulators
• Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Variational f-divergence Minimization
• Understanding Variational Inference in Function-Space
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Wasserstein Variational Inference
• Stochastic Maximum Likelihood Optimization via Hypernetworks
• A Parsimonious Tour of Bayesian Model Uncertainty
• Latent space projection predictive inference
• Generate High Resolution Images With Generative Variational Autoencoder
• Kernel Implicit Variational Inference
• Adversarially Approximated Autoencoder for Image Generation and Manipulation
• Bidirectional Generative Modeling Using Adversarial Gradient Estimation
• Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference
• Variational Inference with Continuously-Indexed Normalizing Flows
• Posterior Meta-Replay for Continual Learning
• Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation
• Importance Weighted Hierarchical Variational Inference
• Doubly Semi-Implicit Variational Inference
• Autoregressive Score Matching
• Variational Inference over Non-differentiable Cardiac Simulators using Bayesian Optimization
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• Amortised Learning by Wake-Sleep
• Variationally Inferred Sampling Through a Refined Bound for Probabilistic Programs
• Copula-like Variational Inference
• Forward Amortized Inference for Likelihood-Free Variational Marginalization
• Statistical Guarantees for Transformation Based Models with Applications to Implicit Variational Inference
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• Efficient Semi-Implicit Variational Inference
• Thompson Sampling via Local Uncertainty
• Adversarial Variational Bayes Methods for Tweedie Compound Poisson Mixed Models
• Posterior Ratio Estimation of Latent Variables
• Stochastic Bayesian Neural Networks
• Imitation with Neural Density Models
• Dual Adversarial Variational Embedding for Robust Recommendation