Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling

Abstract

The estimation of mutual information (MI) or conditional mutual information (CMI) from a set of samples is a long-standing problem. A recent line of work in this area has leveraged the approximation power of artificial neural networks and has shown improvements over conventional methods. One important challenge in this new approach is the need to obtain, given the original dataset, a different set where the samples are distributed according to a specific product density function. This is particularly challenging when estimating CMI. In this paper, we introduce a new technique, based on k nearest neighbors (k-NN), to perform the resampling and derive high-confidence concentration bounds for the sample average. Then the technique is employed to train a neural network classifier and the CMI is estimated accordingly. We propose three estimators using this technique and prove their consistency, make a comparison between them and similar approaches in the literature, and experimentally show improvements in estimating the CMI in terms of accuracy and variance of the estimators.

References in corpus (14)

• Estimating Mutual Information
• Deep Learning-Based Communication Over the Air
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• MINE: Mutual Information Neural Estimation
• On Variational Bounds of Mutual Information
• Formal Limitations on the Measurement of Mutual Information
• Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimator
• Understanding the Limitations of Variational Mutual Information Estimators
• Conditional Mutual Information Neural Estimator
• ITENE: Intrinsic Transfer Entropy Neural Estimator
• On the Estimation of Information Measures of Continuous Distributions
• Machine Learning using the Variational Predictive Information Bottleneck with a Validation Set
• Testing for directed information graphs
• Causality Graph of Vehicular Traffic Flow

Cited by in corpus (1)

• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation