Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities

Abstract

Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic integration, approximated, or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy and the Kullback-Leibler divergence of mixtures. We illustrate the versatile method by reporting on our experiments for approximating the Kullback-Leibler divergence between univariate exponential mixtures, Gaussian mixtures, Rayleigh mixtures, and Gamma mixtures.

20 pages, 3 figures

Cited by in corpus (8)

• Stochastic Modelling of Urban Structure
• A series of maximum entropy upper bounds of the differential entropy
• Monte Carlo Information Geometry: The dually flat case
• Using the Softplus Function to Construct Alternative Link Functions in Generalized Linear Models and Beyond
• Thermodynamics of exponential Kolmogorov-Nagumo averages
• On the design of autonomous agents from multiple data sources
• Sedimentation path theory for mass-polydisperse colloidal systems
• The Bregman chord divergence