Demystifying MMD GANs

Abstract

We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

arXiv commentPublished at ICLR 2018: https://openreview.net/forum?id=r1lUOzWCW

References in corpus (20)

• Scikit-learn: Machine Learning in Python
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Equivalence of distance-based and RKHS-based statistics in hypothesis testing
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Generative Moment Matching Networks
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Better Mixing via Deep Representations
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Training generative neural networks via Maximum Mean Discrepancy optimization
• Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
• Towards the Automatic Anime Characters Creation with Generative Adversarial Networks
• Do GANs actually learn the distribution? An empirical study
• On integral probability metrics, ϕ-divergences and binary classification
• Learning Generative Models with Sinkhorn Divergences
• Comparison of Maximum Likelihood and GAN-based training of Real NVPs
• A Test of Relative Similarity For Model Selection in Generative Models
• B-tests: Low Variance Kernel Two-Sample Tests
• Distributional Adversarial Networks

Cited by in corpus (51)

• Training Generative Adversarial Networks with Limited Data
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• How to GAN LHC Events
• Classification Accuracy Score for Conditional Generative Models
• Deep Knockoffs
• VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers
• Diagnosing and Enhancing VAE Models
• On Accurate Evaluation of GANs for Language Generation
• Prescribed Generative Adversarial Networks
• COT-GAN: Generating Sequential Data via Causal Optimal Transport
• Attention2AngioGAN: Synthesizing Fluorescein Angiography from Retinal Fundus Images using Generative Adversarial Networks
• Distributionally Robust Bayesian Optimization
• Neural Language Generation: Formulation, Methods, and Evaluation
• Robust Estimation and Generative Adversarial Nets
• Review of end-to-end speech synthesis technology based on deep learning
• Mimicry: Towards the Reproducibility of GAN Research
• Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning
• Implicit competitive regularization in GANs
• Sliced Kernelized Stein Discrepancy
• Weakly Supervised Generative Network for Multiple 3D Human Pose Hypotheses
• Bridging the gap between paired and unpaired medical image translation
• Latent Translation: Crossing Modalities by Bridging Generative Models
• Conditional out-of-sample generation for unpaired data using trVAE
• Estimating Certain Integral Probability Metric (IPM) is as Hard as Estimating under the IPM
• Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
• Generative Adversarial Nets for Robust Scatter Estimation: A Proper Scoring Rule Perspective
• MixerGAN: An MLP-Based Architecture for Unpaired Image-to-Image Translation
• Training Generative Reversible Networks
• Learning Implicit Generative Models with the Method of Learned Moments
• First Order Generative Adversarial Networks
• Unsupervised Shadow Removal Using Target Consistency Generative Adversarial Network
• xAI-GAN: Enhancing Generative Adversarial Networks via Explainable AI Systems
• Mask Based Unsupervised Content Transfer
• Using Simulated Data to Generate Images of Climate Change
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds
• Neural Network Generalization: The impact of camera parameters
• Reciprocal Adversarial Learning via Characteristic Functions
• Adaptive Weighted Discriminator for Training Generative Adversarial Networks
• Training Generative Adversarial Networks in One Stage
• Information Condensing Active Learning
• Unsupervised Multi-Domain Multimodal Image-to-Image Translation with Explicit Domain-Constrained Disentanglement
• Large-Scale Optimal Transport via Adversarial Training with Cycle-Consistency
• Synthesis and Edition of Ultrasound Images via Sketch Guided Progressive Growing GANs
• Learning Manifold Implicitly via Explicit Heat-Kernel Learning
• Online Kernel based Generative Adversarial Networks
• Invexifying Regularization of Non-Linear Least-Squares Problems
• Establishing an Evaluation Metric to Quantify Climate Change Image Realism
• One Shot Audio to Animated Video Generation
• One-element Batch Training by Moving Window
• On the Optimization Landscape of Maximum Mean Discrepancy
• Bridging the Visual Gap: Wide-Range Image Blending