Generative Adversarial Networks as Variational Training of Energy Based Models

Abstract

In this paper, we study deep generative models for effective unsupervised learning. We propose VGAN, which works by minimizing a variational lower bound of the negative log likelihood (NLL) of an energy based model (EBM), where the model density $p(\mathbf{x})$ is approximated by a variational distribution $q(\mathbf{x})$ that is easy to sample from. The training of VGAN takes a two step procedure: given $p(\mathbf{x})$, $q(\mathbf{x})$ is updated to maximize the lower bound; $p(\mathbf{x})$ is then updated one step with samples drawn from $q(\mathbf{x})$ to decrease the lower bound. VGAN is inspired by the generative adversarial networks (GANs), where $p(\mathbf{x})$ corresponds to the discriminator and $q(\mathbf{x})$ corresponds to the generator, but with several notable differences. We hence name our model variational GANs (VGANs). VGAN provides a practical solution to training deep EBMs in high dimensional space, by eliminating the need of MCMC sampling. From this view, we are also able to identify causes to the difficulty of training GANs and propose viable solutions. \footnote{Experimental code is available at https://github.com/Shuangfei/vgan}

arXiv commentUnder review at ICLR 2017

References in corpus (5)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• ADADELTA: An Adaptive Learning Rate Method
• Energy-based Generative Adversarial Network
• Deep Structured Energy Based Models for Anomaly Detection
• Deep Directed Generative Models with Energy-Based Probability Estimation

Cited by in corpus (11)

• How to Train Your Energy-Based Models
• A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources
• Pedestrian-Synthesis-GAN: Generating Pedestrian Data in Real Scene and Beyond
• SmartSeed: Smart Seed Generation for Efficient Fuzzing
• Boosting Deep Learning Risk Prediction with Generative Adversarial Networks for Electronic Health Records
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE
• Bounds all around: training energy-based models with bidirectional bounds
• To Relieve Your Headache of Training an MRF, Take AdVIL
• Bayesian Cycle-Consistent Generative Adversarial Networks via Marginalizing Latent Sampling
• Learning Proposals for Practical Energy-Based Regression