Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities

Abstract

The Jeffreys divergence is a renown symmetrization of the oriented Kullback-Leibler divergence broadly used in information sciences. Since the Jeffreys divergence between Gaussian mixture models is not available in closed-form, various techniques with pros and cons have been proposed in the literature to either estimate, approximate, or lower and upper bound this divergence. In this paper, we propose a simple yet fast heuristic to approximate the Jeffreys divergence between two univariate Gaussian mixtures with arbitrary number of components. Our heuristic relies on converting the mixtures into pairs of dually parameterized probability densities belonging to an exponential family. In particular, we consider the versatile polynomial exponential family densities, and design a divergence to measure in closed-form the goodness of fit between a Gaussian mixture and its polynomial exponential density approximation. This goodness-of-fit divergence is a generalization of the Hyvärinen divergence used to estimate models with computationally intractable normalizers. It allows us to perform model selection by choosing the orders of the polynomial exponential densities used to approximate the mixtures. We demonstrate experimentally that our heuristic to approximate the Jeffreys divergence improves by several orders of magnitude the computational time of stochastic Monte Carlo estimations while approximating reasonably well the Jeffreys divergence, specially when the mixtures have a very small number of modes. Besides, our mixture-to-exponential family conversion techniques may prove useful in other settings.

arXiv comment43 pages

References in corpus (16)

• On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
• An elementary introduction to information geometry
• Statistical exponential families: A digest with flash cards
• Moment Varieties of Gaussian Mixtures
• $k$-MLE: A fast algorithm for learning statistical mixture models
• Relative Loss Bounds for On-line Density Estimation with the Exponential Family of Distributions
• Exponential Varieties
• On the symmetrical Kullback-Leibler Jeffreys centroids
• Offline Reinforcement Learning with Fisher Divergence Critic Regularization
• Maximum Number of Modes of Gaussian Mixtures
• Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities
• A series of maximum entropy upper bounds of the differential entropy
• Monte Carlo Information Geometry: The dually flat case
• Inference in Hybrid Bayesian Networks Using Mixtures of Gaussians
• Variational approximations using Fisher divergence
• Change Detection in Multivariate Datastreams: Likelihood and Detectability Loss

Cited by in corpus (1)

• Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities