Conditional Mutual Information Neural Estimator

Abstract

Several recent works in communication systems have proposed to leverage the power of neural networks in the design of encoders and decoders. In this approach, these blocks can be tailored to maximize the transmission rate based on aggregated samples from the channel. Motivated by the fact that, in many communication schemes, the achievable transmission rate is determined by a conditional mutual information term, this paper focuses on neural-based estimators for this information-theoretic quantity. Our results are based on variational bounds for the KL-divergence and, in contrast to some previous works, we provide a mathematically rigorous lower bound. However, additional challenges with respect to the unconditional mutual information emerge due to the presence of a conditional density function which we address here.

arXiv commentTo be presented at ICASSP 2020

References in corpus (13)

• Representation Learning with Contrastive Predictive Coding
• Estimating Mutual Information
• The information bottleneck method
• Learning deep representations by mutual information estimation and maximization
• Deep Learning-Based Communication Over the Air
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• MINE: Mutual Information Neural Estimation
• Entropy and mutual information in models of deep neural networks
• Formal Limitations on the Measurement of Mutual Information
• CCMI : Classifier based Conditional Mutual Information Estimation
• Transfer-Entropy-Regularized Markov Decision Processes
• Testing for directed information graphs

Cited by in corpus (2)

• Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling
• On Mutual Information Neural Estimation for Localization