Data-Efficient Mutual Information Neural Estimator
arXiv:1905.03319
Abstract
Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications. Recent work, MINE (Belghazi et al. 2018), focused on estimating tight variational lower bounds of MI using neural networks, but assumed unlimited supply of samples to prevent overfitting. In real world applications, data is not always available at a surplus. In this work, we focus on improving data efficiency and propose a Data-Efficient MINE Estimator (DEMINE), by developing a relaxed predictive MI lower bound that can be estimated at higher data efficiency by orders of magnitudes. The predictive MI lower bound also enables us to develop a new meta-learning approach using task augmentation, Meta-DEMINE, to improve generalization of the network and further boost estimation accuracy empirically. With improved data-efficiency, our estimators enables statistical testing of dependency at practical dataset sizes. We demonstrate the effectiveness of our estimators on synthetic benchmarks and a real world fMRI data, with application of inter-subject correlation analysis.
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Prototypical Networks for Few-shot Learning
- Estimating Mutual Information
- Understanding deep learning requires rethinking generalization
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Bayesian Model-Agnostic Meta-Learning
- Formal Limitations on the Measurement of Mutual Information