On Bregman Distances and Divergences of Probability Measures

Abstract

The paper introduces scaled Bregman distances of probability distributions which admit non-uniform contributions of observed events. They are introduced in a general form covering not only the distances of discrete and continuous stochastic observations, but also the distances of random processes and signals. It is shown that the scaled Bregman distances extend not only the classical ones studied in the previous literature, but also the information divergence and the related wider class of convex divergences of probability measures. An information processing theorem is established too, but only in the sense of invariance w.r.t. statistically sufficient transformations and not in the sense of universal monotonicity. Pathological situations where coding can increase the classical Bregman distance are illustrated by a concrete example. In addition to the classical areas of application of the Bregman distances and convex divergences such as recognition, classification, learning and evaluation of proximity of various features and signals, the paper mentions a new application in 3D-exploratory data analysis. Explicit expressions for the scaled Bregman distances are obtained in general exponential families, with concrete applications in the binomial, Poisson and Rayleigh families, and in the families of exponential processes such as the Poisson and diffusion processes including the classical examples of the Wiener process and geometric Brownian motion.

12 two-column pages, 2 figures

Cited by in corpus (13)

• Extropy: Complementary Dual of Entropy
• Bayesian Inference with Posterior Regularization and applications to Infinite Latent SVMs
• Deformed Statistics Kullback-Leibler Divergence Minimization within a Scaled Bregman Framework
• A Scale-invariant Generalization of the Rényi Entropy, Associated Divergences and their Optimizations under Tsallis' Nonextensive Framework
• A precise bare simulation approach to the minimization of some distances. Foundations
• Assessing fluctuations of long-memory environmental variables based on the robustified dynamic Orlicz risk
• On conformal divergences and their population minimizers
• Some distance bounds of branching processes and their diffusion limits
• Scaled Bregman divergences in a Tsallis scenario
• An Objective Prior from a Scoring Rule
• Optimal transport with some directed distances
• BreGMN: scaled-Bregman Generative Modeling Networks
• A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions