Sliced Score Matching: A Scalable Approach to Density and Score Estimation

Abstract

Score matching is a popular method for estimating unnormalized statistical models. However, it has been so far limited to simple, shallow models or low-dimensional data, due to the difficulty of computing the Hessian of log-density functions. We show this difficulty can be mitigated by projecting the scores onto random vectors before comparing them. This objective, called sliced score matching, only involves Hessian-vector products, which can be easily implemented using reverse-mode automatic differentiation. Therefore, sliced score matching is amenable to more complex models and higher dimensional data compared to score matching. Theoretically, we prove the consistency and asymptotic normality of sliced score matching estimators. Moreover, we demonstrate that sliced score matching can be used to learn deep score estimators for implicit distributions. In our experiments, we show sliced score matching can learn deep energy-based models effectively, and can produce accurate score estimates for applications such as variational inference with implicit distributions and training Wasserstein Auto-Encoders.

arXiv commentUAI 2019

References in corpus (9)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• NICE: Non-linear Independent Components Estimation
• Functional Variational Bayesian Neural Networks
• Variational Inference using Implicit Distributions
• Density Estimation in Infinite Dimensional Exponential Families
• Deep Energy Estimator Networks
• Gradient-free Hamiltonian Monte Carlo with Efficient Kernel Exponential Families
• Estimating the Hessian by Back-propagating Curvature

Cited by in corpus (29)

• Diffusion Models in Vision: A Survey
• Generative Modeling by Estimating Gradients of the Data Distribution
• Improved Techniques for Training Score-Based Generative Models
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Learning to Efficiently Sample from Diffusion Probabilistic Models
• Maximum Likelihood Training of Score-Based Diffusion Models
• WaveGrad: Estimating Gradients for Waveform Generation
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• Ab-initio study of interacting fermions at finite temperature with neural canonical transformation
• Review of end-to-end speech synthesis technology based on deep learning
• Sliced Kernelized Stein Discrepancy
• Energy-Based Imitation Learning
• Score-based Generative Modeling in Latent Space
• Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability
• Mutual Information Gradient Estimation for Representation Learning
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Kernel Stein Generative Modeling
• Density Ratio Estimation via Infinitesimal Classification
• Efficient Learning of Generative Models via Finite-Difference Score Matching
• Bi-level Score Matching for Learning Energy-based Latent Variable Models
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
• On Out-of-distribution Detection with Energy-based Models
• Autoregressive Score Matching
• Nonparametric Score Estimators
• Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
• Active Slices for Sliced Stein Discrepancy
• Imitation with Neural Density Models