MMD GAN: Towards Deeper Understanding of Moment Matching Network

Abstract

Generative moment matching network (GMMN) is a deep generative model that differs from Generative Adversarial Network (GAN) by replacing the discriminator in GAN with a two-sample test based on kernel maximum mean discrepancy (MMD). Although some theoretical guarantees of MMD have been studied, the empirical performance of GMMN is still not as competitive as that of GAN on challenging and large benchmark datasets. The computational efficiency of GMMN is also less desirable in comparison with GAN, partially due to its requirement for a rather large batch size during the training. In this paper, we propose to improve both the model expressiveness of GMMN and its computational efficiency by introducing adversarial kernel learning techniques, as the replacement of a fixed Gaussian kernel in the original GMMN. The new approach combines the key ideas in both GMMN and GAN, hence we name it MMD GAN. The new distance measure in MMD GAN is a meaningful loss that enjoys the advantage of weak topology and can be optimized via gradient descent with relatively small batch sizes. In our evaluation on multiple benchmark datasets, including MNIST, CIFAR- 10, CelebA and LSUN, the performance of MMD-GAN significantly outperforms GMMN, and is competitive with other representative GAN works.

In the Proceedings of Thirty-first Annual Conference on Neural Information Processing Systems (NIPS 2017)

References in corpus (11)

• Improved Training of Wasserstein GANs
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Energy-based Generative Adversarial Network
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Adversarially Learned Inference
• Kernel Mean Embedding of Distributions: A Review and Beyond
• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
• Generative Moment Matching Networks
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning
• Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy

Cited by in corpus (146)

• Recent advances and clinical applications of deep learning in medical image analysis
• How Generative Adversarial Networks and Their Variants Work: An Overview
• Differentiable Learning of Quantum Circuit Born Machine
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Wasserstein Auto-Encoders
• How to GAN LHC Events
• Demystifying MMD GANs
• MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks
• Towards the Automatic Anime Characters Creation with Generative Adversarial Networks
• Yao.jl: Extensible, Efficient Framework for Quantum Algorithm Design
• Deep Knockoffs
• A survey on GANs for computer vision: Recent research, analysis and taxonomy
• Robust Unsupervised Domain Adaptation for Neural Networks via Moment Alignment
• Simulation of electron-proton scattering events by a Feature-Augmented and Transformed Generative Adversarial Network (FAT-GAN)
• Learning Generative Models with Sinkhorn Divergences
• Pedestrian-Synthesis-GAN: Generating Pedestrian Data in Real Scene and Beyond
• Stabilizing Generative Adversarial Networks: A Survey
• PI-VAE: Physics-Informed Variational Auto-Encoder for stochastic differential equations
• On gradient regularizers for MMD GANs
• Maximum Mean Discrepancy Gradient Flow
• Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
• Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators
• Moment Matching for Multi-Source Domain Adaptation
• Multi-adversarial Faster-RCNN for Unrestricted Object Detection
• COT-GAN: Generating Sequential Data via Causal Optimal Transport
• HIGAN: Cosmic Neutral Hydrogen with Generative Adversarial Networks
• Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Representation, learning, and planning algorithms for geometric task and motion planning
• Learning Generative Models across Incomparable Spaces
• Boosting Deep Learning Risk Prediction with Generative Adversarial Networks for Electronic Health Records
• The Shape of Data: Intrinsic Distance for Data Distributions
• DP-MERF: Differentially Private Mean Embeddings with Random Features for Practical Privacy-Preserving Data Generation
• Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space with Softplus Loss
• Calibrated Reliable Regression using Maximum Mean Discrepancy
• Causal Generative Domain Adaptation Networks
• Metrizing Weak Convergence with Maximum Mean Discrepancies
• Wasserstein-2 Generative Networks
• A survey of machine learning-based physics event generation
• Benchmarking Regression Methods: A comparison with CGAN
• Minibatch optimal transport distances; analysis and applications
• Some Theoretical Insights into Wasserstein GANs
• Heterogeneous Domain Adaptation with Adversarial Neural Representation Learning: Experiments on E-Commerce and Cybersecurity
• Conditional Born machine for Monte Carlo event generation
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
• Fair Generative Modeling via Weak Supervision
• Implicit competitive regularization in GANs
• Exchangeable deep neural networks for set-to-set matching and learning
• Efficient Generation of Structured Objects with Constrained Adversarial Networks
• TraDE: Transformers for Density Estimation
• Switchable Whitening for Deep Representation Learning
• Linear Time Sinkhorn Divergences using Positive Features
• Quantitatively Evaluating GANs With Divergences Proposed for Training
• A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Latent Translation: Crossing Modalities by Bridging Generative Models
• cGANs with Multi-Hinge Loss
• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images
• Human Action Generation with Generative Adversarial Networks
• Learning deep kernels for exponential family densities
• Likelihood Estimation for Generative Adversarial Networks
• Non-parametric estimation of Jensen-Shannon Divergence in Generative Adversarial Network training
• Two Sample Testing in High Dimension via Maximum Mean Discrepancy
• Towards Robust GAN-generated Image Detection: a Multi-view Completion Representation
• Estimating Barycenters of Measures in High Dimensions
• Learning Kernel Tests Without Data Splitting
• The energy distance for ensemble and scenario reduction
• Training Generative Adversarial Networks via Primal-Dual Subgradient Methods: A Lagrangian Perspective on GAN
• Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents
• Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets
• Training Generative Reversible Networks
• Classification Logit Two-sample Testing by Neural Networks
• Sequential Attention GAN for Interactive Image Editing
• Maximum Mean Discrepancy for Generalization in the Presence of Distribution and Missingness Shift
• Neural SDEs as Infinite-Dimensional GANs
• Self-Supervised GANs with Label Augmentation
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Learning Implicit Generative Models with the Method of Learned Moments
• Two-sample Testing Using Deep Learning
• Novel Human-Object Interaction Detection via Adversarial Domain Generalization
• PEARL: Data Synthesis via Private Embeddings and Adversarial Reconstruction Learning
• Designing GANs: A Likelihood Ratio Approach
• Generalized Energy Based Models
• Generative Ratio Matching Networks
• Privacy Enhancing Machine Learning via Removal of Unwanted Dependencies
• DISSECT: Disentangled Simultaneous Explanations via Concept Traversals
• Refining Deep Generative Models via Discriminator Gradient Flow
• Generative Adversarial Mapping Networks
• Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions
• Adaptive Weighted Discriminator for Training Generative Adversarial Networks
• Dodging DeepFake Detection via Implicit Spatial-Domain Notch Filtering
• Point Cloud Completion by Learning Shape Priors
• Reciprocal Adversarial Learning via Characteristic Functions
• Mixability of Integral Losses: a Key to Efficient Online Aggregation of Functional and Probabilistic Forecasts
• Improved BiGAN training with marginal likelihood equalization
• CUDA: Contradistinguisher for Unsupervised Domain Adaptation
• Universal Hypothesis Testing with Kernels: Asymptotically Optimal Tests for Goodness of Fit
• Kernel-Guided Training of Implicit Generative Models with Stability Guarantees
• Variational Transport: A Convergent Particle-BasedAlgorithm for Distributional Optimization
• Learning Manifold Implicitly via Explicit Heat-Kernel Learning
• Discrepancy-based Inference for Intractable Generative Models using Quasi-Monte Carlo
• Bayesian Cycle-Consistent Generative Adversarial Networks via Marginalizing Latent Sampling
• not-so-BigGAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution
• Pareto GAN: Extending the Representational Power of GANs to Heavy-Tailed Distributions
• Risk Bounds for Unsupervised Cross-Domain Mapping with IPMs
• On the Convergence of Gradient Descent in GANs: MMD GAN As a Gradient Flow
• Kernel Conditional Density Operators
• Efficient and Accurate Gradients for Neural SDEs
• Tessellated Wasserstein Auto-Encoders
• Asymptotically Optimal One- and Two-Sample Testing with Kernels
• Topic Modeling with Wasserstein Autoencoders
• KernelNet: A Data-Dependent Kernel Parameterization for Deep Generative Modeling
• Making Method of Moments Great Again? -- How can GANs learn distributions
• Geodesics in fibered latent spaces: A geometric approach to learning correspondences between conditions
• Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
• Unsupervised Object Matching for Relational Data
• On reproduction of On the regularization of Wasserstein GANs
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• Scalable Bayesian Nonparametric Clustering and Classification
• Generative Modeling by Inclusive Neural Random Fields with Applications in Image Generation and Anomaly Detection
• MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning
• Semi-Supervised Learning with IPM-based GANs: an Empirical Study
• Exponentially Consistent Kernel Two-Sample Tests
• Moment Matching Training for Neural Machine Translation: A Preliminary Study
• LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds
• Implicit Manifold Learning on Generative Adversarial Networks
• On the Optimization Landscape of Maximum Mean Discrepancy
• Forward Operator Estimation in Generative Models with Kernel Transfer Operators
• Scalable Kernel Learning via the Discriminant Information
• Unsupervised Dense Deformation Embedding Network for Template-Free Shape Correspondence
• Data-driven Regularized Inference Privacy
• Minimax Nonparametric Two-sample Test under Smoothing
• Deep Kernel Learning for Clustering
• Coulomb Autoencoders
• Pooling Regularized Graph Neural Network for fMRI Biomarker Analysis
• High-Dimensional Non-Parametric Density Estimation in Mixed Smooth Sobolev Spaces
• The Fourier Discrepancy Function
• Semi-Supervised Self-Growing Generative Adversarial Networks for Image Recognition
• Unsupervised Program Synthesis for Images By Sampling Without Replacement
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks