Flow Contrastive Estimation of Energy-Based Models

Abstract

This paper studies a training method to jointly estimate an energy-based model and a flow-based model, in which the two models are iteratively updated based on a shared adversarial value function. This joint training method has the following traits. (1) The update of the energy-based model is based on noise contrastive estimation, with the flow model serving as a strong noise distribution. (2) The update of the flow model approximately minimizes the Jensen-Shannon divergence between the flow model and the data distribution. (3) Unlike generative adversarial networks (GAN) which estimates an implicit probability distribution defined by a generator model, our method estimates two explicit probabilistic distributions on the data. Using the proposed method we demonstrate a significant improvement on the synthesis quality of the flow model, and show the effectiveness of unsupervised feature learning by the learned energy-based model. Furthermore, the proposed training method can be easily adapted to semi-supervised learning. We achieve competitive results to the state-of-the-art semi-supervised learning methods.

References in corpus (24)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• WaveNet: A Generative Model for Raw Audio
• Spectral Normalization for Generative Adversarial Networks
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Generative Modeling by Estimating Gradients of the Data Distribution
• Density estimation using Real NVP
• Auxiliary Deep Generative Models
• Deep Structured Energy Based Models for Anomaly Detection
• Stacked What-Where Auto-encoders
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• A Theory of Generative ConvNet
• Implicit Generation and Generalization in Energy-Based Models
• Deep Directed Generative Models with Energy-Based Probability Estimation
• Deep Energy Estimator Networks
• Discrete Flows: Invertible Generative Models of Discrete Data
• Introspective Classification with Convolutional Nets
• Adversarial Contrastive Estimation
• Adversarial Fisher Vectors for Unsupervised Representation Learning

Cited by in corpus (3)

• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• How to Train Your Energy-Based Model for Regression
• Imitation with Neural Density Models