Generative Moment Matching Networks

Abstract

We consider the problem of learning deep generative models from data. We formulate a method that generates an independent sample via a single feedforward pass through a multilayer perceptron, as in the recently proposed generative adversarial networks (Goodfellow et al., 2014). Training a generative adversarial network, however, requires careful optimization of a difficult minimax program. Instead, we utilize a technique from statistical hypothesis testing known as maximum mean discrepancy (MMD), which leads to a simple objective that can be interpreted as matching all orders of statistics between a dataset and samples from the model, and can be trained by backpropagation. We further boost the performance of this approach by combining our generative network with an auto-encoder network, using MMD to learn to generate codes that can then be decoded to produce samples. We show that the combination of these techniques yields excellent generative models compared to baseline approaches as measured on MNIST and the Toronto Face Database.

References in corpus (13)

• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Practical Bayesian Optimization of Machine Learning Algorithms
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Generative Stochastic Networks Trainable by Backprop
• Better Mixing via Deep Representations
• Generalized Denoising Auto-Encoders as Generative Models
• Neural Variational Inference and Learning in Belief Networks
• Training generative neural networks via Maximum Mean Discrepancy optimization
• Input Warping for Bayesian Optimization of Non-stationary Functions
• A Generative Process for Sampling Contractive Auto-Encoders
• FastMMD: Ensemble of Circular Discrepancy for Efficient Two-Sample Test

Cited by in corpus (215)

• An Introduction to Variational Autoencoders
• Deep Visual Domain Adaptation: A Survey
• NIPS 2016 Tutorial: Generative Adversarial Networks
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Unsupervised Image-to-Image Translation Networks
• Kernel Mean Embedding of Distributions: A Review and Beyond
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• A note on the evaluation of generative models
• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
• How Generative Adversarial Networks and Their Variants Work: An Overview
• InfoVAE: Information Maximizing Variational Autoencoders
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• Adversarial Autoencoders
• Differentially Private Generative Adversarial Network
• Differentiable Learning of Quantum Circuit Born Machine
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Triple Generative Adversarial Nets
• Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning
• Stabilizing Training of Generative Adversarial Networks through Regularization
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Removal of Batch Effects using Distribution-Matching Residual Networks
• Wasserstein Auto-Encoders
• How to GAN LHC Events
• Demystifying MMD GANs
• Deep Residual Correction Network for Partial Domain Adaptation
• GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders
• Deep Knockoffs
• Coupled Generative Adversarial Networks
• Generative Image Modeling Using Spatial LSTMs
• Learning and Inference on Generative Adversarial Quantum Circuits
• Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation
• A-NICE-MC: Adversarial Training for MCMC
• Large-Scale Optimal Transport and Mapping Estimation
• Stabilizing Generative Adversarial Networks: A Survey
• Variational Auto-encoded Deep Gaussian Processes
• Deep Manifold Traversal: Changing Labels with Convolutional Features
• Approximation and Convergence Properties of Generative Adversarial Learning
• Implicit Maximum Likelihood Estimation
• On gradient regularizers for MMD GANs
• Maximum Mean Discrepancy Gradient Flow
• Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• Deep Signature Transforms
• Moment Matching for Multi-Source Domain Adaptation
• Prescribed Generative Adversarial Networks
• Stacked Generative Adversarial Networks
• Conditional Sig-Wasserstein GANs for Time Series Generation
• A Selective Overview of Deep Learning
• An Improved Evaluation Framework for Generative Adversarial Networks
• Graph Normalizing Flows
• A Convex Duality Framework for GANs
• Optimal transport mapping via input convex neural networks
• Understanding GANs: the LQG Setting
• Statistical Inference for Generative Models with Maximum Mean Discrepancy
• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Conditional Generative Moment-Matching Networks
• The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models
• How Well Generative Adversarial Networks Learn Distributions
• Probabilistic Autoencoder
• Rates of Estimation of Optimal Transport Maps using Plug-in Estimators via Barycentric Projections
• Multi-source Domain Adaptation for Text-independent Forensic Speaker Recognition
• Domain Adaptation as a Problem of Inference on Graphical Models
• Memorization Precedes Generation: Learning Unsupervised GANs with Memory Networks
• Generative machine learning methods for multivariate ensemble post-processing
• On the capacity of deep generative networks for approximating distributions
• McGan: Mean and Covariance Feature Matching GAN
• Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification
• Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• Causal Generative Neural Networks
• Robust Estimation and Generative Adversarial Nets
• Cluster Alignment with a Teacher for Unsupervised Domain Adaptation
• Calibrated Reliable Regression using Maximum Mean Discrepancy
• A Spectral Energy Distance for Parallel Speech Synthesis
• Causal Generative Domain Adaptation Networks
• Smoothness and Stability in GANs
• Neural Autoregressive Distribution Estimation
• Rethink Maximum Mean Discrepancy for Domain Adaptation
• An error analysis of generative adversarial networks for learning distributions
• Distributionally Robust Optimization and Generalization in Kernel Methods
• Kernel Two-Sample Tests in High Dimension: Interplay Between Moment Discrepancy and Dimension-and-Sample Orders
• Conditional Born machine for Monte Carlo event generation
• Knowledge distillation via adaptive instance normalization
• Generative Adversarial Networks (GANs): What it can generate and What it cannot?
• Stein Neural Sampler
• Auto-Differentiating Linear Algebra
• Minimum Stein Discrepancy Estimators
• Implicit competitive regularization in GANs
• Kernel Change-point Detection with Auxiliary Deep Generative Models
• MONK -- Outlier-Robust Mean Embedding Estimation by Median-of-Means
• Learning the Redundancy-free Features for Generalized Zero-Shot Object Recognition
• MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
• Reproducing Kernel Hilbert Space, Mercer's Theorem, Eigenfunctions, Nyström Method, and Use of Kernels in Machine Learning: Tutorial and Survey
• Conditional out-of-sample generation for unpaired data using trVAE
• Energy-Based Imitation Learning
• Sampling-based speech parameter generation using moment-matching networks
• Learning to Write Stylized Chinese Characters by Reading a Handful of Examples
• ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech
• Non-parametric estimation of Jensen-Shannon Divergence in Generative Adversarial Network training
• Triple Generative Adversarial Networks
• Associative Adversarial Networks
• Zero-Shot Semantic Segmentation
• High Fidelity Face Manipulation with Extreme Poses and Expressions
• Learning Unsupervised Word Mapping by Maximizing Mean Discrepancy
• Towards Automatic Image Editing: Learning to See another You
• Gaussian Copula Variational Autoencoders for Mixed Data
• Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors
• A Unified Joint Maximum Mean Discrepancy for Domain Adaptation
• Learning Kernel Tests Without Data Splitting
• Training Generative Adversarial Networks via Primal-Dual Subgradient Methods: A Lagrangian Perspective on GAN
• Classification Logit Two-sample Testing by Neural Networks
• Gaussian mixture models with Wasserstein distance
• Neural SDEs as Infinite-Dimensional GANs
• Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance
• Maximum Mean Discrepancy for Generalization in the Presence of Distribution and Missingness Shift
• Bias-Free FedGAN: A Federated Approach to Generate Bias-Free Datasets
• Improving Conditional Coverage via Orthogonal Quantile Regression
• Deep Divergence Learning
• Learning to Match Distributions for Domain Adaptation
• Normalized Wasserstein Distance for Mixture Distributions with Applications in Adversarial Learning and Domain Adaptation
• Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis
• Rescuing neural spike train models from bad MLE
• Quantile-Quantile Embedding for Distribution Transformation and Manifold Embedding with Ability to Choose the Embedding Distribution
• Synthetic-Neuroscore: Using A Neuro-AI Interface for Evaluating Generative Adversarial Networks
• Wasserstein-Wasserstein Auto-Encoders
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Human Motion Analysis with Deep Metric Learning
• Deep Structured Generative Models
• First Order Generative Adversarial Networks
• Neural Tangent Kernel Maximum Mean Discrepancy
• Deep Generative Learning via Variational Gradient Flow
• Generative Ratio Matching Networks
• Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis
• A Pragmatic Look at Deep Imitation Learning
• Privacy Enhancing Machine Learning via Removal of Unwanted Dependencies
• Learning Implicit Generative Models with Theoretical Guarantees
• Learning Implicit Generative Models by Teaching Explicit Ones
• Deep Dimension Reduction for Supervised Representation Learning
• An explainability study of the constant Q cepstral coefficient spoofing countermeasure for automatic speaker verification
• Multi-Source Domain Adaptation for Text Classification via DistanceNet-Bandits
• GANs with Variational Entropy Regularizers: Applications in Mitigating the Mode-Collapse Issue
• SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentation
• KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support
• Self-balanced Learning For Domain Generalization
• Phase reconstruction from amplitude spectrograms based on von-Mises-distribution deep neural network
• DPD-InfoGAN: Differentially Private Distributed InfoGAN
• Faster Unbalanced Optimal Transport: Translation invariant Sinkhorn and 1-D Frank-Wolfe
• Universal Hypothesis Testing with Kernels: Asymptotically Optimal Tests for Goodness of Fit
• Generative Image Modeling using Style and Structure Adversarial Networks
• A Mathematical Framework for Learning Probability Distributions
• Latent Space Refinement for Deep Generative Models
• Double cycle-consistent generative adversarial network for unsupervised conditional generation
• Recursive Training for Zero-Shot Semantic Segmentation
• Efficient and Accurate Gradients for Neural SDEs
• Kernel Conditional Density Operators
• ThumbNet: One Thumbnail Image Contains All You Need for Recognition
• Discrepancy-based Inference for Intractable Generative Models using Quasi-Monte Carlo
• Kernel-Guided Training of Implicit Generative Models with Stability Guarantees
• Semi-Supervised Translation with MMD Networks
• Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey
• Exemplar-based synthesis of geology using kernel discrepancies and generative neural networks
• Risk Bounds for Unsupervised Cross-Domain Mapping with IPMs
• A Note on Optimizing Distributions using Kernel Mean Embeddings
• Inferential Wasserstein Generative Adversarial Networks
• On the Convergence of Gradient Descent in GANs: MMD GAN As a Gradient Flow
• Generative Learning With Euler Particle Transport
• A Neuro-AI Interface for Evaluating Generative Adversarial Networks
• Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification
• Deep Lifetime Clustering
• DISCO Nets: DISsimilarity COefficient Networks
• Removal of Batch Effects using Generative Adversarial Networks
• GANs with Conditional Independence Graphs: On Subadditivity of Probability Divergences
• Convergence of Gaussian-smoothed optimal transport distance with sub-gamma distributions and dependent samples
• f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning
• Deep Learning on Attributed Graphs: A Journey from Graphs to Their Embeddings and Back
• Kernel Mean Matching for Content Addressability of GANs
• Fitting summary statistics of neural data with a differentiable spiking network simulator
• A Mean-Field Theory for Kernel Alignment with Random Features in Generative and Discriminative Models
• Quasi-random sampling for multivariate distributions via generative neural networks
• Unsupervised Object Matching for Relational Data
• Reparameterized Variational Divergence Minimization for Stable Imitation
• AReS and MaRS - Adversarial and MMD-Minimizing Regression for SDEs
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• Online Kernel based Generative Adversarial Networks
• Post Selection Inference with Incomplete Maximum Mean Discrepancy Estimator
• Easy High-Dimensional Likelihood-Free Inference
• Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization
• Complexity Controlled Generative Adversarial Networks
• An Iterative Closest Points Approach to Neural Generative Models
• Generative networks as inverse problems with fractional wavelet scattering networks
• Exponentially Consistent Kernel Two-Sample Tests
• Wasserstein GAN Can Perform PCA
• Deep Learning generated observations of galaxy clusters from dark-matter-only simulations
• Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning
• Data-driven Regularized Inference Privacy
• Variational Autoencoded Regression: High Dimensional Regression of Visual Data on Complex Manifold
• A moment-matching metric for latent variable generative models
• Hidden Biases in Unreliable News Detection Datasets
• On the Optimization Landscape of Maximum Mean Discrepancy
• FoCL: Feature-Oriented Continual Learning for Generative Models
• DWMD: Dimensional Weighted Orderwise Moment Discrepancy for Domain-specific Hidden Representation Matching
• Implicit Generative Copulas
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• SupMMD: A Sentence Importance Model for Extractive Summarization using Maximum Mean Discrepancy
• Irregular Convolutional Auto-Encoder on Point Clouds
• Multivariate time-series modeling with generative neural networks
• The Benefits of Pairwise Discriminators for Adversarial Training
• Improve SGD Training via Aligning Mini-batches
• Compressive Learning of Generative Networks
• NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation
• How many moments does MMD compare?
• Coulomb Autoencoders
• Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics
• Replicating Active Appearance Model by Generator Network