Minimum Stein Discrepancy Estimators

Abstract

When maximum likelihood estimation is infeasible, one often turns to score matching, contrastive divergence, or minimum probability flow to obtain tractable parameter estimates. We provide a unifying perspective of these techniques as minimum Stein discrepancy estimators, and use this lens to design new diffusion kernel Stein discrepancy (DKSD) and diffusion score matching (DSM) estimators with complementary strengths. We establish the consistency, asymptotic normality, and robustness of DKSD and DSM estimators, then derive stochastic Riemannian gradient descent algorithms for their efficient optimisation. The main strength of our methodology is its flexibility, which allows us to design estimators with desirable properties for specific models at hand by carefully selecting a Stein discrepancy. We illustrate this advantage for several challenging problems for score matching, such as non-smooth, heavy-tailed or light-tailed densities.

Accepted for publication at NeurIPS 2019

References in corpus (8)

• Generative Moment Matching Networks
• Training generative neural networks via Maximum Mean Discrepancy optimization
• Approximate Bayesian computation with the Wasserstein distance
• Interpretation and Generalization of Score Matching
• Statistical Inference for Generative Models with Maximum Mean Discrepancy
• Riemannian Stein Variational Gradient Descent for Bayesian Inference
• Stein Point Markov Chain Monte Carlo
• Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE

Cited by in corpus (5)

• Characterizations of non-normalized discrete probability distributions and their application in statistics
• Exponential Family Estimation via Adversarial Dynamics Embedding
• Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents
• Minimum $L^q$-distance estimators for non-normalized parametric models
• A Computationally Efficient Method for Learning Exponential Family Distributions