Interpretation and Generalization of Score Matching

Abstract

Score matching is a recently developed parameter learning method that is particularly effective to complicated high dimensional density models with intractable partition functions. In this paper, we study two issues that have not been completely resolved for score matching. First, we provide a formal link between maximum likelihood and score matching. Our analysis shows that score matching finds model parameters that are more robust with noisy training data. Second, we develop a generalization of score matching. Based on this generalization, we further demonstrate an extension of score matching to models of discrete data.

Appears in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI2009)

Cited by in corpus (23)

• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Speed Limits for Macroscopic Transitions
• Gradient Estimators for Implicit Models
• Maximum Likelihood Training of Score-Based Diffusion Models
• Minimum Probability Flow Learning
• Deep Energy Estimator Networks
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• Score Function Features for Discriminative Learning: Matrix and Tensor Framework
• Characterizations of non-normalized discrete probability distributions and their application in statistics
• Minimum Stein Discrepancy Estimators
• Score-based Generative Modeling in Latent Space
• A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation
• Learning deep kernels for exponential family densities
• Boltzmann machines and energy-based models
• Information criteria for non-normalized models
• Bi-level Score Matching for Learning Energy-based Latent Variable Models
• Asymptotic Efficiency of Deterministic Estimators for Discrete Energy-Based Models: Ratio Matching and Pseudolikelihood
• Estimating Density Models with Truncation Boundaries using Score Matching
• Graph-based Composite Local Bregman Divergences on Discrete Sample Spaces
• Fisher Auto-Encoders
• Interaction Models and Generalized Score Matching for Compositional Data
• Denoising Score Matching with Random Fourier Features
• A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models