Telescoping Density-Ratio Estimation
arXiv:2006.12204
Abstract
Density-ratio estimation via classification is a cornerstone of unsupervised learning. It has provided the foundation for state-of-the-art methods in representation learning and generative modelling, with the number of use-cases continuing to proliferate. However, it suffers from a critical limitation: it fails to accurately estimate ratios p/q for which the two densities differ significantly. Empirically, we find this occurs whenever the KL divergence between p and q exceeds tens of nats. To resolve this limitation, we introduce a new framework, telescoping density-ratio estimation (TRE), that enables the estimation of ratios between highly dissimilar densities in high-dimensional spaces. Our experiments demonstrate that TRE can yield substantial improvements over existing single-ratio methods for mutual information estimation, representation learning and energy-based modelling.
References in corpus (34)
- Distributed Representations of Words and Phrases and their Compositionality
- Representation Learning with Contrastive Predictive Coding
- Spectral Normalization for Generative Adversarial Networks
- Self-Attention Generative Adversarial Networks
- An Overview of Multi-Task Learning in Deep Neural Networks
- Learning deep representations by mutual information estimation and maximization
- Generative Modeling by Estimating Gradients of the Data Distribution
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- A Learned Representation For Artistic Style
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- A note on the evaluation of generative models
- Modulating early visual processing by language
- Learning in Implicit Generative Models
- Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
- On Variational Bounds of Mutual Information
- Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
- Monte Carlo Gradient Estimation in Machine Learning
- A Mutual Information Maximization Perspective of Language Representation Learning
- NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport
- Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
- Formal Limitations on the Measurement of Mutual Information
- Understanding the Limitations of Variational Mutual Information Estimators
- Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
- VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
- Autoregressive Energy Machines
- Residual Energy-Based Models for Text
- Dynamic Likelihood-free Inference via Ratio Estimation (DIRE)
- Fair Generative Modeling via Weak Supervision
- Boosted Generative Models
- Training Deep Energy-Based Models with f-Divergence Minimization
- Analysis of Noise Contrastive Estimation from the Perspective of Asymptotic Variance
- Variational Noise-Contrastive Estimation
- Energy-Based Processes for Exchangeable Data
- Noise contrastive estimation: asymptotics, comparison with MC-MLE