Neural Methods for Point-wise Dependency Estimation
arXiv:2006.05553
Abstract
Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables. However, MI is an aggregate statistic and cannot be used to measure point-wise dependency between different events. In this work, instead of estimating the expected dependency, we focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We show that we can naturally obtain PD when we are optimizing MI neural variational bounds. However, optimizing these bounds is challenging due to its large variance in practice. To address this issue, we develop two methods (free of optimizing MI variational bounds): Probabilistic Classifier and Density-Ratio Fitting. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Efficient Estimation of Word Representations in Vector Space
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- A Simple Framework for Contrastive Learning of Visual Representations
- Representation Learning with Contrastive Predictive Coding
- Estimating Mutual Information
- Learning deep representations by mutual information estimation and maximization
- Momentum Contrast for Unsupervised Visual Representation Learning
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Contrastive Multiview Coding
- MINE: Mutual Information Neural Estimation
- On Mutual Information Maximization for Representation Learning
- Learning Factorized Multimodal Representations
- Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
- A Mutual Information Maximization Perspective of Language Representation Learning
- Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
- Understanding the Limitations of Variational Mutual Information Estimators