KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support

Abstract

We study the gradient flow for a relaxed approximation to the Kullback-Leibler (KL) divergence between a moving source and a fixed target distribution. This approximation, termed the KALE (KL approximate lower-bound estimator), solves a regularized version of the Fenchel dual problem defining the KL over a restricted class of functions. When using a Reproducing Kernel Hilbert Space (RKHS) to define the function class, we show that the KALE continuously interpolates between the KL and the Maximum Mean Discrepancy (MMD). Like the MMD and other Integral Probability Metrics, the KALE remains well defined for mutually singular distributions. Nonetheless, the KALE inherits from the limiting KL a greater sensitivity to mismatch in the support of the distributions, compared with the MMD. These two properties make the KALE gradient flow particularly well suited when the target distribution is supported on a low-dimensional manifold. Under an assumption of sufficient smoothness of the trajectories, we show the global convergence of the KALE flow. We propose a particle implementation of the flow given initial samples from the source and the target distribution, which we use to empirically confirm the KALE's properties.

References in corpus (17)

• Rényi Divergence and Kullback-Leibler Divergence
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Generative Moment Matching Networks
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• Training generative neural networks via Maximum Mean Discrepancy optimization
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• Sample Complexity of Sinkhorn divergences
• Interpolating between Optimal Transport and MMD using Sinkhorn Divergences
• Stein Variational Gradient Descent as Gradient Flow
• Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
• On the Discrimination-Generalization Tradeoff in GANs
• On the optimal estimation of probability measures in weak and strong topologies
• Analysis of Langevin Monte Carlo via convex optimization
• Efficient constrained sampling via the mirror-Langevin algorithm
• Statistical Estimation of the Poincar{é} constant and Application to Sampling Multimodal Distributions
• Sinkhorn Barycenter via Functional Gradient Descent

Cited by in corpus (1)

• $(f,\Gamma )$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics