Tighter Variational Representations of f-Divergences via Restriction to Probability Measures

Abstract

We show that the variational representations for f-divergences currently used in the literature can be tightened. This has implications to a number of methods recently proposed based on this representation. As an example application we use our tighter representation to derive a general f-divergence estimator based on two i.i.d. samples and derive the dual program for this estimator that performs well empirically. We also point out a connection between our estimator and MMD.

arXiv commentICML2012

References in corpus (1)

• Estimating divergence functionals and the likelihood ratio by convex risk minimization

Cited by in corpus (8)

• On Mutual Information Maximization for Representation Learning
• Estimating time-dependent entropy production from non-equilibrium trajectories
• Mutual Information Scaling for Tensor Network Machine Learning
• $(f,\Gamma )$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics
• Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation
• Variational Representations and Neural Network Estimation of Rényi Divergences
• On the Theoretical Equivalence of Several Trade-Off Curves Assessing Statistical Proximity
• Moreau-Yosida $f$-divergences