CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information

Abstract

Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.

Accepted by the 37th International Conference on Machine Learing (ICML2020)

Cited by in corpus (12)

• FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders
• Auto IV: Counterfactual Prediction via Automatic Instrumental Variable Decomposition
• Learning Invariant Inter-pixel Correlations for Superpixel Generation
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Learning to Diversify for Single Domain Generalization
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion
• Disentangling representations of retinal images with generative models
• Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization
• Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environment
• Selective Regression Under Fairness Criteria
• On Study of Mutual Information and its Estimation Methods
• Unsupervised Domain Adaptation for Dysarthric Speech Detection via Domain Adversarial Training and Mutual Information Minimization