Generalized Energy Based Models

Abstract

We introduce the Generalized Energy Based Model (GEBM) for generative modelling. These models combine two trained components: a base distribution (generally an implicit model), which can learn the support of data with low intrinsic dimension in a high dimensional space; and an energy function, to refine the probability mass on the learned support. Both the energy function and base jointly constitute the final model, unlike GANs, which retain only the base distribution (the "generator"). GEBMs are trained by alternating between learning the energy and the base. We show that both training stages are well-defined: the energy is learned by maximising a generalized likelihood, and the resulting energy-based loss provides informative gradients for learning the base. Samples from the posterior on the latent space of the trained model can be obtained via MCMC, thus finding regions in this space that produce better quality samples. Empirically, the GEBM samples on image-generation tasks are of much better quality than those from the learned generator alone, indicating that all else being equal, the GEBM will outperform a GAN of the same complexity. When using normalizing flows as base measures, GEBMs succeed on density modelling tasks, returning comparable performance to direct maximum likelihood of the same networks.

References in corpus (42)

• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Pixel Recurrent Neural Networks
• Improved Training of Wasserstein GANs
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Variational Inference with Normalizing Flows
• Continual Learning Through Synaptic Intelligence
• Density estimation using Real NVP
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Large Scale Adversarial Representation Learning
• MADE: Masked Autoencoder for Distribution Estimation
• On Convergence and Stability of GANs
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
• Underdamped Langevin MCMC: A non-asymptotic analysis
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Gradient descent GAN optimization is locally stable
• Density Estimation in Infinite Dimensional Exponential Families
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• Approximation and Convergence Properties of Generative Adversarial Learning
• Deep Compressed Sensing
• On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
• LOGAN: Latent Optimisation for Generative Adversarial Networks
• On gradient regularizers for MMD GANs
• Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling
• Stochastic subgradient method converges at the rate $O(k^{-1/4})$ on weakly convex functions
• Flows for simultaneous manifold learning and density estimation
• Autoregressive Quantile Networks for Generative Modeling
• Efficient Algorithms for Smooth Minimax Optimization
• Discriminator Rejection Sampling
• Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
• Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space with Softplus Loss
• Smoothness and Stability in GANs
• Kernel Exponential Family Estimation via Doubly Dual Embedding
• Discriminator optimal transport
• Exponential Family Estimation via Adversarial Dynamics Embedding
• Training Deep Energy-Based Models with f-Divergence Minimization

Cited by in corpus (3)

• A case for new neural network smoothness constraints
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Particle Dynamics for Learning EBMs