TCMI: a non-parametric mutual-dependence estimator for multivariate continuous distributions

Abstract

The identification of relevant features, i.e., the driving variables that determine a process or the properties of a system, is an essential part of the analysis of data sets with a large number of variables. A mathematical rigorous approach to quantifying the relevance of these features is mutual information. Mutual information determines the relevance of features in terms of their joint mutual dependence to the property of interest. However, mutual information requires as input probability distributions, which cannot be reliably estimated from continuous distributions such as physical quantities like lengths or energies. Here, we introduce total cumulative mutual information (TCMI), a measure of the relevance of mutual dependences that extends mutual information to random variables of continuous distribution based on cumulative probability distributions. TCMI is a non-parametric, robust, and deterministic measure that facilitates comparisons and rankings between feature sets with different cardinality. The ranking induced by TCMI allows for feature selection, i.e., the identification of variable sets that are nonlinear statistically related to a property of interest, taking into account the number of data samples as well as the cardinality of the set of variables. We evaluate the performance of our measure with simulated data, compare its performance with similar multivariate-dependence measures, and demonstrate the effectiveness of our feature-selection method on a set of standard data sets and a typical scenario in materials science.

arXiv comment28 pages, 8 figures, 8 tables

References in corpus (10)

• Estimating Mutual Information
• Measuring and testing dependence by correlation of distances
• A Review of Feature Selection Methods Based on Mutual Information
• Big Data of Materials Science - Critical Role of the Descriptor
• SISSO: a compressed-sensing method for identifying the best low-dimensional descriptor in an immensity of offered candidates
• Learning physical descriptors for materials science by compressed sensing
• Discovering Reliable Approximate Functional Dependencies
• Self-consistent method for density estimation
• On Quantifying Dependence: A Framework for Developing Interpretable Measures
• Simple stopping criteria for information theoretic feature selection