Practical and Consistent Estimation of f-Divergences

Abstract

The estimation of an f-divergence between two probability distributions based on samples is a fundamental problem in statistics and machine learning. Most works study this problem under very weak assumptions, in which case it is provably hard. We consider the case of stronger structural assumptions that are commonly satisfied in modern machine learning, including representation learning and generative modelling with autoencoder architectures. Under these assumptions we propose and study an estimator that can be easily implemented, works well in high dimensions, and enjoys faster rates of convergence. We verify the behavior of our estimator empirically in both synthetic and real-data experiments, and discuss its direct implications for total correlation, entropy, and mutual information estimation.

arXiv commentAccepted to the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada

References in corpus (8)

• Representation Learning with Contrastive Predictive Coding
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Isolating Sources of Disentanglement in Variational Autoencoders
• Rényi Divergence Variational Inference
• Ensemble estimation of multivariate f-divergence
• Avoiding Latent Variable Collapse With Generative Skip Models
• Multivariate f-Divergence Estimation With Confidence