Likelihood Ratio Exponential Families
arXiv:2012.15480
Abstract
The exponential family is well known in machine learning and statistical physics as the maximum entropy distribution subject to a set of observed constraints, while the geometric mixture path is common in MCMC methods such as annealed importance sampling. Linking these two ideas, recent work has interpreted the geometric mixture path as an exponential family of distributions to analyze the thermodynamic variational objective (TVO). We extend these likelihood ratio exponential families to include solutions to rate-distortion (RD) optimization, the information bottleneck (IB) method, and recent rate-distortion-classification approaches which combine RD and IB. This provides a common mathematical framework for understanding these methods via the conjugate duality of exponential families and hypothesis testing. Further, we collect existing results to provide a variational representation of intermediate RD or TVO distributions as a minimizing an expectation of KL divergences. This solution also corresponds to a size-power tradeoff using the likelihood ratio test and the Neyman Pearson lemma. In thermodynamic integration bounds such as the TVO, we identify the intermediate distribution whose expected sufficient statistics match the log partition function.
arXiv commentNeurIPS Workshop on Deep Learning through Information Geometry
References in corpus (11)
- Rényi Divergence and Kullback-Leibler Divergence
- Equilibrium free energy differences from nonequilibrium measurements: a master equation approach
- Deep Variational Information Bottleneck
- On Variational Bounds of Mutual Information
- Chernoff information of exponential families
- A family of statistical symmetric divergences based on Jensen's inequality
- Interpretations of Renyi Entropies And Divergences
- The Information Bottleneck EM Algorithm
- TherML: Thermodynamics of Machine Learning
- Evaluating Lossy Compression Rates of Deep Generative Models
- DEMI: Discriminative Estimator of Mutual Information