Resampling methods for parameter-free and robust feature selection with mutual information

Abstract

Combining the mutual information criterion with a forward feature selection strategy offers a good trade-off between optimality of the selected feature subset and computation time. However, it requires to set the parameter(s) of the mutual information estimator and to determine when to halt the forward procedure. These two choices are difficult to make because, as the dimensionality of the subset increases, the estimation of the mutual information becomes less and less reliable. This paper proposes to use resampling methods, a K-fold cross-validation and the permutation test, to address both issues. The resampling methods bring information about the variance of the estimator, information which can then be used to automatically set the parameter and to calculate a threshold to stop the forward procedure. The procedure is illustrated on a synthetic dataset as well as on real-world examples.

References in corpus (3)

• Estimating Mutual Information
• Mutual information for the selection of relevant variables in spectrometric nonlinear modelling
• Representation of Functional Data in Neural Networks

Cited by in corpus (8)

• Combining complex networks and data mining: why and how
• Expert System Based On Neural-Fuzzy Rules for Thyroid Diseases Diagnosis
• Advances in Feature Selection with Mutual Information
• Measuring the Discrepancy between Conditional Distributions: Methods, Properties and Applications
• Simple stopping criteria for information theoretic feature selection
• A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition
• Feature Relevance Determination for Ordinal Regression in the Context of Feature Redundancies and Privileged Information
• InfoNEAT: Information Theory-based NeuroEvolution of Augmenting Topologies for Side-channel Analysis