Variational Representations and Neural Network Estimation of Rényi Divergences

Abstract

We derive a new variational formula for the Rényi family of divergences, $R_{\alpha }(Q\Vert P)$, between probability measures $Q$ and $P$. Our result generalizes the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. We further show that this Rényi variational formula holds over a range of function spaces; this leads to a formula for the optimizer under very weak assumptions and is also key in our development of a consistency theory for Rényi divergence estimators. By applying this theory to neural-network estimators, we show that if a neural network family satisfies one of several strengthened versions of the universal approximation property then the corresponding Rényi divergence estimator is consistent. In contrast to density-estimator based methods, our estimators involve only expectations under $Q$ and $P$ and hence are more effective in high dimensional systems. We illustrate this via several numerical examples of neural network estimation in systems of up to 5000 dimensions.

24 pages, 2 figures

References in corpus (4)

• On the Convergence of Adam and Beyond
• On Variational Expressions for Quantum Relative Entropies
• Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions
• Tighter Variational Representations of f-Divergences via Restriction to Probability Measures

Cited by in corpus (1)

• $(f,\Gamma )$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics