On Study of Mutual Information and its Estimation Methods
arXiv:2106.14646
Abstract
The presence of mutual information in the research of deep learning has grown significantly. It has been proven that mutual information can be a good objective function to build a robust deep learning model. Most of the researches utilize estimation methods to approximate the true mutual information. This technical report delivers an extensive study about definitions as well as properties of mutual information. This article then delivers some reviews and current drawbacks of mutual information estimation methods afterward.
References in corpus (8)
- InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Deep Variational Information Bottleneck
- Stochastic Neural Networks for Hierarchical Reinforcement Learning
- On Variational Bounds of Mutual Information
- CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
- Understanding the Limitations of Variational Mutual Information Estimators
- Wasserstein Dependency Measure for Representation Learning