Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models

Abstract

Deep generative models are a class of techniques that train deep neural networks to model the distribution of training samples. Research has fragmented into various interconnected approaches, each of which make trade-offs including run-time, diversity, and architectural restrictions. In particular, this compendium covers energy-based models, variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, in addition to numerous hybrid approaches. These techniques are compared and contrasted, explaining the premises behind each and how they are interrelated, while reviewing current state-of-the-art advances and implementations.

arXiv comment20 pages, 9 figures, will appear in IEEE Transactions on Pattern Analysis and Machine Intelligence

References in corpus (98)

• Auto-Encoding Variational Bayes
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Conditional Generative Adversarial Nets
• Denoising Diffusion Probabilistic Models
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• Self-Attention Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Conditional Image Generation with PixelCNN Decoders
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Generative Adversarial Network in Medical Imaging: A Review
• Wide Residual Networks
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Normalizing Flows: An Introduction and Review of Current Methods
• Zero-Shot Text-to-Image Generation
• Generative Modeling by Estimating Gradients of the Data Distribution
• Energy-based Generative Adversarial Network
• Linformer: Self-Attention with Linear Complexity
• Density estimation using Real NVP
• The relativistic discriminator: a key element missing from standard GAN
• Generating Long Sequences with Sparse Transformers
• Improved Denoising Diffusion Probabilistic Models
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• On Data Augmentation for GAN Training
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• Variational Lossy Autoencoder
• Image Transformer
• Stochastic Adversarial Video Prediction
• Generalized Denoising Auto-Encoders as Generative Models
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Wasserstein Auto-Encoders
• Residual Flows for Invertible Generative Modeling
• A Theory of Generative ConvNet
• IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
• Amortised MAP Inference for Image Super-resolution
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Towards Deeper Understanding of Variational Autoencoding Models
• Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates
• Image Augmentations for GAN Training
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Sylvester Normalizing Flows for Variational Inference
• Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis
• Implicit Generation and Generalization in Energy-Based Models
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• Diagnosing and Enhancing VAE Models
• How to Train Your Energy-Based Models
• PixelVAE: A Latent Variable Model for Natural Images
• Deep Directed Generative Models with Energy-Based Probability Estimation
• A-NICE-MC: Adversarial Training for MCMC
• Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport
• Maximum Entropy Generators for Energy-Based Models
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• The continuous Bernoulli: fixing a pervasive error in variational autoencoders
• Latent Normalizing Flows for Discrete Sequences
• A Spectral Approach to Gradient Estimation for Implicit Distributions
• Emerging Convolutions for Generative Normalizing Flows
• SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
• Deep Energy Estimator Networks
• Discrete Flows: Invertible Generative Models of Discrete Data
• Sum-of-Squares Polynomial Flow
• Monge-Ampère Flow for Generative Modeling
• IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression
• Towards GAN Benchmarks Which Require Generalization
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• The Lipschitz Constant of Self-Attention
• Approximation Capabilities of Neural ODEs and Invertible Residual Networks
• A domain agnostic measure for monitoring and evaluating GANs
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Block Neural Autoregressive Flow
• Cubic-Spline Flows
• On Contrastive Learning for Likelihood-free Inference
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• Autoregressive Energy Machines
• Learnable Explicit Density for Continuous Latent Space and Variational Inference
• Improving the Speed and Quality of GAN by Adversarial Training
• Generative Models as Distributions of Functions
• MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
• Spectral Regularization for Combating Mode Collapse in GANs
• The Variational Gaussian Process
• Gradient-based Adaptive Markov Chain Monte Carlo
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
• Anycost GANs for Interactive Image Synthesis and Editing
• Generative Minimization Networks: Training GANs Without Competition
• Exemplar VAE: Linking Generative Models, Nearest Neighbor Retrieval, and Data Augmentation
• Implicit Normalizing Flows
• A Neural Network MCMC sampler that maximizes Proposal Entropy
• Improved Autoregressive Modeling with Distribution Smoothing

Cited by in corpus (5)

• Denoising diffusion probabilistic models for probabilistic energy forecasting
• Emerging Directions in Geophysical Inversion
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• Improving ensemble extreme precipitation forecasts using generative artificial intelligence
• Dual Variational Autoencoders for Efficient Sim-to-Real Transfer in Low-Cost Robotic Navigation