Blindness of score-based methods to isolated components and mixing proportions
arXiv:2008.10087
Abstract
Statistical tasks such as density estimation and approximate Bayesian inference often involve densities with unknown normalising constants. Score-based methods, including score matching, are popular techniques as they are free of normalising constants. Although these methods enjoy theoretical guarantees, a little-known fact is that they exhibit practical failure modes when the unnormalised distribution of interest has isolated components -- they cannot discover isolated components or identify the correct mixing proportions between components. We demonstrate these findings using simple distributions and present heuristic attempts to address these issues. We hope to bring the attention of theoreticians and practitioners to these issues when developing new algorithms and applications.
References in corpus (8)
- Generative Modeling by Estimating Gradients of the Data Distribution
- Measuring Sample Quality with Stein's Method
- A Kernel Test of Goodness of Fit
- Measuring Sample Quality with Kernels
- On gradient regularizers for MMD GANs
- Gradient Estimators for Implicit Models
- Message Passing Stein Variational Gradient Descent
- A Spectral Approach to Gradient Estimation for Implicit Distributions