Learning in Implicit Generative Models

Abstract

Generative adversarial networks (GANs) provide an algorithmic framework for constructing generative models with several appealing properties: they do not require a likelihood function to be specified, only a generating procedure; they provide samples that are sharp and compelling; and they allow us to harness our knowledge of building highly accurate neural network classifiers. Here, we develop our understanding of GANs with the aim of forming a rich view of this growing area of machine learning---to build connections to the diverse set of statistical thinking on this topic, of which much can be gained by a mutual exchange of ideas. We frame GANs within the wider landscape of algorithms for learning in implicit generative models--models that only specify a stochastic procedure with which to generate data--and relate these ideas to modelling problems in related fields, such as econometrics and approximate Bayesian computation. We develop likelihood-free inference methods and highlight hypothesis testing as a principle for learning in implicit generative models, using which we are able to derive the objective function used by GANs, and many other related objectives. The testing viewpoint directs our focus to the general problem of density ratio estimation. There are four approaches for density ratio estimation, one of which is a solution using classifiers to distinguish real from generated data. Other approaches such as divergence minimisation and moment matching have also been explored in the GAN literature, and we synthesise these views to form an understanding in terms of the relationships between them and the wider literature, highlighting avenues for future exploration and cross-pollination.

References in corpus (6)

• Energy-based Generative Adversarial Network
• On integral probability metrics, ϕ-divergences and binary classification
• Improved generator objectives for GANs
• Generative Adversarial Nets from a Density Ratio Estimation Perspective
• Adversarial Message Passing For Graphical Models
• Task Specific Adversarial Cost Function

Cited by in corpus (33)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Generative Adversarial Residual Pairwise Networks for One Shot Learning
• Variational Inference using Implicit Distributions
• Improved generator objectives for GANs
• Prescribed Generative Adversarial Networks
• Statistical Inference for Generative Models with Maximum Mean Discrepancy
• Learning to Draw Samples with Amortized Stein Variational Gradient Descent
• Rates of Estimation of Optimal Transport Maps using Plug-in Estimators via Barycentric Projections
• Information bottleneck through variational glasses
• TF-Replicator: Distributed Machine Learning for Researchers
• Adversarial Message Passing For Graphical Models
• Generative Adversarial Nets for Robust Scatter Estimation: A Proper Scoring Rule Perspective
• Diagnosing model misspecification and performing generalized Bayes' updates via probabilistic classifiers
• Image Quality Assessment Techniques Show Improved Training and Evaluation of Autoencoder Generative Adversarial Networks
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder
• Learning Implicit Generative Models with Theoretical Guarantees
• Deep Generative Learning via Variational Gradient Flow
• Generalized Sliced Distances for Probability Distributions
• Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
• Uncertainty-Aware Deep Classifiers using Generative Models
• Importance Weighted Hierarchical Variational Inference
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Generating Unobserved Alternatives
• Generative Learning With Euler Particle Transport
• Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds
• ORQVIZ: Visualizing High-Dimensional Landscapes in Variational Quantum Algorithms
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Robust model training and generalisation with Studentising flows
• Kernel Mean Matching for Content Addressability of GANs
• Out-of-Sample Testing for GANs
• Continual Density Ratio Estimation in an Online Setting
• Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics