Learning in Implicit Generative Models

Abstract

Generative adversarial networks (GANs) provide an algorithmic framework for constructing generative models with several appealing properties: they do not require a likelihood function to be specified, only a generating procedure; they provide samples that are sharp and compelling; and they allow us to harness our knowledge of building highly accurate neural network classifiers. Here, we develop our understanding of GANs with the aim of forming a rich view of this growing area of machine learning---to build connections to the diverse set of statistical thinking on this topic, of which much can be gained by a mutual exchange of ideas. We frame GANs within the wider landscape of algorithms for learning in implicit generative models--models that only specify a stochastic procedure with which to generate data--and relate these ideas to modelling problems in related fields, such as econometrics and approximate Bayesian computation. We develop likelihood-free inference methods and highlight hypothesis testing as a principle for learning in implicit generative models, using which we are able to derive the objective function used by GANs, and many other related objectives. The testing viewpoint directs our focus to the general problem of density ratio estimation. There are four approaches for density ratio estimation, one of which is a solution using classifiers to distinguish real from generated data. Other approaches such as divergence minimisation and moment matching have also been explored in the GAN literature, and we synthesise these views to form an understanding in terms of the relationships between them and the wider literature, highlighting avenues for future exploration and cross-pollination.

References in corpus (12)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Energy-based Generative Adversarial Network
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• A note on the evaluation of generative models
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
• On integral probability metrics, ϕ-divergences and binary classification
• Improved generator objectives for GANs
• Generative Adversarial Nets from a Density Ratio Estimation Perspective
• Likelihood-free inference via classification
• Adversarial Message Passing For Graphical Models
• Task Specific Adversarial Cost Function

Cited by in corpus (118)

• Disentangling by Factorising
• Normalizing Flows for Probabilistic Modeling and Inference
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Mining gold from implicit models to improve likelihood-free inference
• The Born Supremacy: Quantum Advantage and Training of an Ising Born Machine
• Least Squares Generative Adversarial Networks
• Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference
• Denoising Diffusion Implicit Models
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Generative Adversarial Residual Pairwise Networks for One Shot Learning
• Monte Carlo Gradient Estimation in Machine Learning
• DisCo Fever: Robust Networks Through Distance Correlation
• A-NICE-MC: Adversarial Training for MCMC
• Variational Inference using Implicit Distributions
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• Faster Wasserstein Distance Estimation with the Sinkhorn Divergence
• Implicit Maximum Likelihood Estimation
• Generative model benchmarks for superconducting qubits
• Adversarial-Residual-Coarse-Graining: Applying machine learning theory to systematic molecular coarse-graining
• Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality
• Gradient Estimators for Implicit Models
• Improved generator objectives for GANs
• Distribution Matching in Variational Inference
• Prescribed Generative Adversarial Networks
• A Spectral Approach to Gradient Estimation for Implicit Distributions
• Statistical Inference for Generative Models with Maximum Mean Discrepancy
• A Tutorial on Deep Latent Variable Models of Natural Language
• The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models
• Learning to Draw Samples with Amortized Stein Variational Gradient Descent
• Generative models for sampling and phase transition indication in spin systems
• Rates of Estimation of Optimal Transport Maps using Plug-in Estimators via Barycentric Projections
• Likelihood-free MCMC with Amortized Approximate Ratio Estimators
• Generative machine learning methods for multivariate ensemble post-processing
• Advances in Variational Inference
• Semi-Implicit Variational Inference
• Approximate Inference with Amortised MCMC
• Information bottleneck through variational glasses
• On parameter estimation with the Wasserstein distance
• Effective LHC measurements with matrix elements and machine learning
• TF-Replicator: Distributed Machine Learning for Researchers
• Adversarial Variational Optimization of Non-Differentiable Simulators
• Adversarial Message Passing For Graphical Models
• A Review of Learning with Deep Generative Models from Perspective of Graphical Modeling
• Bayesian Neural Ordinary Differential Equations
• Fair Generative Modeling via Weak Supervision
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• On the Implicit Assumptions of GANs
• Statistical guarantees for generative models without domination
• Deep Generative Quantile-Copula Models for Probabilistic Forecasting
• Inference via low-dimensional couplings
• Training Deep Energy-Based Models with f-Divergence Minimization
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Generative Adversarial Nets for Robust Scatter Estimation: A Proper Scoring Rule Perspective
• Variational f-divergence Minimization
• Non-parametric estimation of Jensen-Shannon Divergence in Generative Adversarial Network training
• Diagnosing model misspecification and performing generalized Bayes' updates via probabilistic classifiers
• Maximum Entropy Flow Networks
• Path Integral Sampler: a stochastic control approach for sampling
• Learning Implicit Generative Models with the Method of Learned Moments
• Generative Models as a Data Source for Multiview Representation Learning
• Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions
• Kernel Implicit Variational Inference
• Image Quality Assessment Techniques Show Improved Training and Evaluation of Autoencoder Generative Adversarial Networks
• Variational Inference with Continuously-Indexed Normalizing Flows
• Telescoping Density-Ratio Estimation
• Bidirectional Generative Modeling Using Adversarial Gradient Estimation
• Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder
• Confidence Sets and Hypothesis Testing in a Likelihood-Free Inference Setting
• Learning Implicit Generative Models with Theoretical Guarantees
• Deep Random Splines for Point Process Intensity Estimation of Neural Population Data
• Learning Implicit Generative Models by Teaching Explicit Ones
• Deep Generative Learning via Variational Gradient Flow
• Variational Rejection Sampling
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling
• Variational Inference with Holder Bounds
• Reciprocal Adversarial Learning via Characteristic Functions
• Improved BiGAN training with marginal likelihood equalization
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Uncertainty-Aware Deep Classifiers using Generative Models
• Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
• Permutation Weighting
• Latent Dirichlet Allocation in Generative Adversarial Networks
• Generalized Sliced Distances for Probability Distributions
• Adversarial Constraint Learning for Structured Prediction
• Conditional Generative Adversarial Networks for Emoji Synthesis with Word Embedding Manipulation
• Importance Weighted Hierarchical Variational Inference
• Filtering Variational Objectives
• General Probabilistic Surface Optimization and Log Density Estimation
• Generative Learning With Euler Particle Transport
• Generating Unobserved Alternatives
• Design of Experiments for Model Discrimination Hybridising Analytical and Data-Driven Approaches
• Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds
• A Novel Framework for Selection of GANs for an Application
• Discrepancy-based Inference for Intractable Generative Models using Quasi-Monte Carlo
• Doubly Semi-Implicit Variational Inference
• Semi-Supervised Translation with MMD Networks
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Out-of-Sample Testing for GANs
• ORQVIZ: Visualizing High-Dimensional Landscapes in Variational Quantum Algorithms
• Discriminative Learning of Similarity and Group Equivariant Representations
• Deep PDF: Probabilistic Surface Optimization and Density Estimation
• Kernel Mean Matching for Content Addressability of GANs
• Robust Density Estimation under Besov IPM Losses
• On Open-Universe Causal Reasoning
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• No Representation without Transformation
• PassFlow: Guessing Passwords with Generative Flows
• Robust model training and generalisation with Studentising flows
• Easy High-Dimensional Likelihood-Free Inference
• Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics
• Bridging the Gap Between $f$-GANs and Wasserstein GANs
• Dual Projection Generative Adversarial Networks for Conditional Image Generation
• Continual Density Ratio Estimation in an Online Setting
• Imitation with Neural Density Models
• MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Space
• Implicit Generative Copulas