$(f,\Gamma )$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics

Abstract

We develop a rigorous and general framework for constructing information-theoretic divergences that subsume both $f$-divergences and integral probability metrics (IPMs), such as the $1$-Wasserstein distance. We prove under which assumptions these divergences, hereafter referred to as $(f,\Gamma )$-divergences, provide a notion of `distance' between probability measures and show that they can be expressed as a two-stage mass-redistribution/mass-transport process. The $(f,\Gamma )$-divergences inherit features from IPMs, such as the ability to compare distributions which are not absolutely continuous, as well as from $f$-divergences, namely the strict concavity of their variational representations and the ability to control heavy-tailed distributions for particular choices of $f$. When combined, these features establish a divergence with improved properties for estimation, statistical learning, and uncertainty quantification applications. Using statistical learning as an example, we demonstrate their advantage in training generative adversarial networks (GANs) for heavy-tailed, not-absolutely continuous sample distributions. We also show improved performance and stability over gradient-penalized Wasserstein GAN in image generation.

arXiv comment49 pages

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach
• On integral probability metrics, ϕ-divergences and binary classification
• A Convex Duality Framework for GANs
• Tighter Variational Representations of f-Divergences via Restriction to Probability Measures
• The Inductive Bias of Restricted f-GANs
• Variational Representations and Neural Network Estimation of Rényi Divergences
• KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support
• Formulation and properties of a divergence used to compare probability measures without absolute continuity and its application to uncertainty quantification

Cited by in corpus (3)

• Wasserstein-based fairness interpretability framework for machine learning models
• Designing GANs: A Likelihood Ratio Approach
• Moreau-Yosida $f$-divergences