Deep Directed Generative Models with Energy-Based Probability Estimation

Abstract

Training energy-based probabilistic models is confronted with apparently intractable sums, whose Monte Carlo estimation requires sampling from the estimated probability distribution in the inner loop of training. This can be approximately achieved by Markov chain Monte Carlo methods, but may still face a formidable obstacle that is the difficulty of mixing between modes with sharp concentrations of probability. Whereas an MCMC process is usually derived from a given energy function based on mathematical considerations and requires an arbitrarily long time to obtain good and varied samples, we propose to train a deep directed generative model (not a Markov chain) so that its sampling distribution approximately matches the energy function that is being trained. Inspired by generative adversarial networks, the proposed framework involves training of two models that represent dual views of the estimated probability distribution: the energy function (mapping an input configuration to a scalar energy value) and the generator (mapping a noise vector to a generated configuration), both represented by deep neural networks.

References in corpus (2)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Better Mixing via Deep Representations

Cited by in corpus (43)

• Energy-based Generative Adversarial Network
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Implicit Generation and Generalization in Energy-Based Models
• How to Train Your Energy-Based Models
• Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Maximum Entropy Generators for Energy-Based Models
• Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling
• Activation Maximization Generative Adversarial Nets
• A Distributional Approach to Controlled Text Generation
• Improved Contrastive Divergence Training of Energy Based Models
• Generative Adversarial Networks as Variational Training of Energy Based Models
• A Review of Learning with Deep Generative Models from Perspective of Graphical Modeling
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE
• Exponential Family Estimation via Adversarial Dynamics Embedding
• Unsupervised Learning of Compositional Energy Concepts
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting
• Synergetic Learning Systems: Concept, Architecture, and Algorithms
• Inception Score, Label Smoothing, Gradient Vanishing and -log(D(x)) Alternative
• Boltzmann machines and energy-based models
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• Compositional Visual Generation and Inference with Energy Based Models
• Distributional Reinforcement Learning for Energy-Based Sequential Models
• Bounds all around: training energy-based models with bidirectional bounds
• Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis
• Flow Contrastive Estimation of Energy-Based Models
• Model Based Planning with Energy Based Models
• To Relieve Your Headache of Training an MRF, Take AdVIL
• Energy-Based Models with Applications to Speech and Language Processing
• Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model
• Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
• Deep Consensus Learning
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• A Tale of Three Probabilistic Families: Discriminative, Descriptive and Generative Models
• Deep PDF: Probabilistic Surface Optimization and Density Estimation
• Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Sampling
• Explaining the effects of non-convergent sampling in the training of Energy-Based Models
• Predicting Ground-Level Scene Layout from Aerial Imagery
• Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Representation Learning: A Statistical Perspective
• Learning Proposals for Practical Energy-Based Regression