Statistical Inference for Generative Models with Maximum Mean Discrepancy

Abstract

While likelihood-based inference and its variants provide a statistically efficient and widely applicable approach to parametric inference, their application to models involving intractable likelihoods poses challenges. In this work, we study a class of minimum distance estimators for intractable generative models, that is, statistical models for which the likelihood is intractable, but simulation is cheap. The distance considered, maximum mean discrepancy (MMD), is defined through the embedding of probability measures into a reproducing kernel Hilbert space. We study the theoretical properties of these estimators, showing that they are consistent, asymptotically normal and robust to model misspecification. A main advantage of these estimators is the flexibility offered by the choice of kernel, which can be used to trade-off statistical efficiency and robustness. On the algorithmic side, we study the geometry induced by MMD on the parameter space and use this to introduce a novel natural gradient descent-like algorithm for efficient implementation of these estimators. We illustrate the relevance of our theoretical results on several classes of models including a discrete-time latent Markov process and two multivariate stochastic differential equation models.

References in corpus (27)

• Auto-Encoding Variational Bayes
• Generative Moment Matching Networks
• Learning with a Wasserstein Loss
• Exact simulation of diffusions
• MCMC for doubly-intractable distributions
• Training generative neural networks via Maximum Mean Discrepancy optimization
• Learning in Implicit Generative Models
• Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
• Approximate Bayesian computation with the Wasserstein distance
• Sample Complexity of Sinkhorn divergences
• Theory and Applications of Proper Scoring Rules
• Approximate Bayesian Computation for Astronomical Model Analysis: A Case Study in Galaxy Demographics and Morphological Transformation at High Redshift
• Fast Two-Sample Testing with Analytic Representations of Probability Measures
• Large sample analysis of the median heuristic
• On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
• An equation-free computational approach for extracting population-level behavior from individual-based models of biological dispersal
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• On the optimal estimation of probability measures in weak and strong topologies
• Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC
• Conditional Generative Moment-Matching Networks
• Adaptivity and Computation-Statistics Tradeoffs for Kernel and Distance based High Dimensional Two Sample Testing
• A Pessimistic Approximation for the Fisher Information Measure
• Approximate inference with Wasserstein gradient flows
• Wasserstein Training of Boltzmann Machines
• Optimal transport natural gradient for statistical manifolds with continuous sample space
• On the High-dimensional Power of Linear-time Kernel Two-Sample Testing under Mean-difference Alternatives
• Strictly Proper Kernel Scoring Rules and Divergences with an Application to Kernel Two-Sample Hypothesis Testing

Cited by in corpus (16)

• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence
• Metrizing Weak Convergence with Maximum Mean Discrepancies
• Minimum Stein Discrepancy Estimators
• Mean-Field Neural ODEs via Relaxed Optimal Control
• Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents
• Generalized Bayesian Likelihood-Free Inference
• Neural SDEs as Infinite-Dimensional GANs
• Optimal quantisation of probability measures using maximum mean discrepancy
• Estimation of copulas via Maximum Mean Discrepancy
• Nonconvex sparse regularization for deep neural networks and its optimality
• Robust Bayesian Inference for Discrete Outcomes with the Total Variation Distance
• Discrepancy-based Inference for Intractable Generative Models using Quasi-Monte Carlo
• Approximate Bayesian Computation with Path Signatures
• Consistent Long-Term Forecasting of Ergodic Dynamical Systems