On Unifying Deep Generative Models

Abstract

Deep generative models have achieved impressive success in recent years. Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), as emerging families for generative model learning, have largely been considered as two distinct paradigms and received extensive independent studies respectively. This paper aims to establish formal connections between GANs and VAEs through a new formulation of them. We interpret sample generation in GANs as performing posterior inference, and show that GANs and VAEs involve minimizing KL divergences of respective posterior and inference distributions with opposite directions, extending the two learning phases of classic wake-sleep algorithm, respectively. The unified view provides a powerful tool to analyze a diverse set of existing model variants, and enables to transfer techniques across research lines in a principled way. For example, we apply the importance weighting method in VAE literatures for improved GAN learning, and enhance VAEs with an adversarial mechanism that leverages generated samples. Experiments show generality and effectiveness of the transferred techniques.

Polished and extended content over the ICLR conference version: https://openreview.net/pdf?id=rylSzl-R-

Cited by in corpus (26)

• Improving Variational Autoencoder with Deep Feature Consistent and Generative Adversarial Training
• Adversarially Regularized Autoencoders
• Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites
• Generative Latent Flow
• Symmetric Variational Autoencoder and Connections to Adversarial Learning
• A Review of Learning with Deep Generative Models from Perspective of Graphical Modeling
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• Decomposing Textual Information For Style Transfer
• Automatic Article Commenting: the Task and Dataset
• Deep Reasoning Networks: Thinking Fast and Slow
• Variational Inference: A Unified Framework of Generative Models and Some Revelations
• Approximating Constraint Manifolds Using Generative Models for Sampling-Based Constrained Motion Planning
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• InfoCatVAE: Representation Learning with Categorical Variational Autoencoders
• Concept-Oriented Deep Learning: Generative Concept Representations
• Adversarially Approximated Autoencoder for Image Generation and Manipulation
• Learning to Match Distributions for Domain Adaptation
• Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders
• Bidirectional Generative Modeling Using Adversarial Gradient Estimation
• Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference
• Information Bottleneck Constrained Latent Bidirectional Embedding for Zero-Shot Learning
• A Mathematical Framework for Learning Probability Distributions
• Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds
• Capturing the diversity of biological tuning curves using generative adversarial networks
• Unaligned Image-to-Image Translation by Learning to Reweight
• Gradient Boosted Normalizing Flows