Training generative neural networks via Maximum Mean Discrepancy optimization

Abstract

We consider training a deep neural network to generate samples from an unknown distribution given i.i.d. data. We frame learning as an optimization minimizing a two-sample test statistic---informally speaking, a good generator network produces samples that cause a two-sample test to fail to reject the null hypothesis. As our two-sample test statistic, we use an unbiased estimate of the maximum mean discrepancy, which is the centerpiece of the nonparametric kernel two-sample test proposed by Gretton et al. (2012). We compare to the adversarial nets framework introduced by Goodfellow et al. (2014), in which learning is a two-player game between a generator network and an adversarial discriminator network, both trained to outwit the other. From this perspective, the MMD statistic plays the role of the discriminator. In addition to empirical comparisons, we prove bounds on the generalization error incurred by optimizing the empirical MMD.

10 pages, to appear in Uncertainty in Artificial Intelligence (UAI) 2015

Cited by in corpus (59)

• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Kernel Mean Embedding of Distributions: A Review and Beyond
• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
• Generative Moment Matching Networks
• Removal of Batch Effects using Distribution-Matching Residual Networks
• Wasserstein Auto-Encoders
• Demystifying MMD GANs
• Deep Knockoffs
• Approximation and Convergence Properties of Generative Adversarial Learning
• Adversarial-Residual-Coarse-Graining: Applying machine learning theory to systematic molecular coarse-graining
• On gradient regularizers for MMD GANs
• Maximum Mean Discrepancy Gradient Flow
• Graph Pooling via Coarsened Graph Infomax
• Deep Signature Transforms
• Statistical Inference for Generative Models with Maximum Mean Discrepancy
• Generative machine learning methods for multivariate ensemble post-processing
• On the capacity of deep generative networks for approximating distributions
• Learning Generative Models across Incomparable Spaces
• Minimax Robust Detection: Classic Results and Recent Advances
• Data Augmentation and Classification of Sea-Land Clutter for Over-the-Horizon Radar Using AC-VAEGAN
• Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence
• Metrizing Weak Convergence with Maximum Mean Discrepancies
• Conditional Born machine for Monte Carlo event generation
• Distributionally Robust Optimization and Generalization in Kernel Methods
• Minimum Stein Discrepancy Estimators
• Kernel Change-point Detection with Auxiliary Deep Generative Models
• MONK -- Outlier-Robust Mean Embedding Estimation by Median-of-Means
• Audio-visual cross-modality knowledge transfer for machine learning-based in-situ monitoring in laser additive manufacturing
• Conditional out-of-sample generation for unpaired data using trVAE
• Estimating Barycenters of Measures in High Dimensions
• Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents
• Maximum Mean Discrepancy for Generalization in the Presence of Distribution and Missingness Shift
• Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance
• Adaptive Object Detection with Dual Multi-Label Prediction
• Wasserstein-Wasserstein Auto-Encoders
• Estimation of copulas via Maximum Mean Discrepancy
• A Pragmatic Look at Deep Imitation Learning
• Deep Generative Learning via Variational Gradient Flow
• Multi-Source Domain Adaptation for Text Classification via DistanceNet-Bandits
• Universal Hypothesis Testing with Kernels: Asymptotically Optimal Tests for Goodness of Fit
• Imitation Learning of Neural Spatio-Temporal Point Processes
• KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support
• Generalized Sliced Distances for Probability Distributions
• Max-Margin Deep Generative Models for (Semi-)Supervised Learning
• Discrepancy-based Inference for Intractable Generative Models using Quasi-Monte Carlo
• Learning Diverse Policies with Soft Self-Generated Guidance
• Measure Transport with Kernel Stein Discrepancy
• On the Convergence of Gradient Descent in GANs: MMD GAN As a Gradient Flow
• Kernel Conditional Density Operators
• Kernel-Guided Training of Implicit Generative Models with Stability Guarantees
• Removal of Batch Effects using Generative Adversarial Networks
• Stable Distribution Alignment Using the Dual of the Adversarial Distance
• Bi-Dimensional Feature Alignment for Cross-Domain Object Detection
• Improving Generative Adversarial Networks with Local Coordinate Coding
• Normalized Diversification
• Irregular Convolutional Auto-Encoder on Point Clouds
• The Benefits of Pairwise Discriminators for Adversarial Training
• Coulomb Autoencoders
• Implicit Generative Copulas