Taming numerical imprecision by adapting the KL divergence to negative probabilities

Abstract

The Kullback-Leibler (KL) divergence is frequently used in data science. For discrete distributions on large state spaces, approximations of probability vectors may result in a few small negative entries, rendering the KL divergence undefined. We address this problem by introducing a parameterized family of substitute divergence measures, the shifted KL (sKL) divergence measures. Our approach is generic and does not increase the computational overhead. We show that the sKL divergence shares important theoretical properties with the KL divergence and discuss how its shift parameters should be chosen. If Gaussian noise is added to a probability vector, we prove that the average sKL divergence converges to the KL divergence for small enough noise. We also show that our method solves the problem of negative entries in an application from computational oncology, the optimization of Mutual Hazard Networks for cancer progression using tensor-train approximations.

11 pages, 6 figures, minor clarifications, published version

References in corpus (12)

• Rényi Divergence and Kullback-Leibler Divergence
• The Infrared Behavior of QCD Green's Functions - Confinement, Dynamical Symmetry Breaking, and Hadrons as Relativistic Bound States
• On Tensors, Sparsity, and Nonnegative Factorizations
• Alternating minimal energy methods for linear systems in higher dimensions. Part II: Faster algorithm and application to nonsymmetric systems
• Alternating minimal energy methods for linear systems in higher dimensions. Part I: SPD systems
• Gluon spectral functions and transport coefficients in Yang--Mills theory
• The entropic prior for distributions with positive and negative values
• Newton-Based Optimization for Kullback-Leibler Nonnegative Tensor Factorizations
• A test on analytic continuation of thermal imaginary-time data
• Approximate probability distributions of the master equation
• Bayesian inference of non-positive spectral functions in quantum field theory
• Estimating the spectrum in computed tomography via Kullback-Leibler divergence constrained optimization