A Debiased MDI Feature Importance Measure for Random Forests

Abstract

Tree ensembles such as Random Forests have achieved impressive empirical success across a wide variety of applications. To understand how these models make predictions, people routinely turn to feature importance measures calculated from tree ensembles. It has long been known that Mean Decrease Impurity (MDI), one of the most widely used measures of feature importance, incorrectly assigns high importance to noisy features, leading to systematic bias in feature selection. In this paper, we address the feature selection bias of MDI from both theoretical and methodological perspectives. Based on the original definition of MDI by Breiman et al. for a single tree, we derive a tight non-asymptotic bound on the expected bias of MDI importance of noisy features, showing that deep trees have higher (expected) feature selection bias than shallow ones. However, it is not clear how to reduce the bias of MDI using its existing analytical expression. We derive a new analytical expression for MDI, and based on this new expression, we are able to propose a debiased MDI feature importance measure using out-of-bag samples, called MDI-oob. For both the simulated data and a genomic ChIP dataset, MDI-oob achieves state-of-the-art performance in feature selection from Random Forests for both deep and shallow trees.

arXiv commentNeurIPS'19. The first two authors contributed equally to this paper

References in corpus (6)

• Scikit-learn: Machine Learning in Python
• Interpretable machine learning: definitions, methods, and applications
• Consistency of random forests
• Consistent Individualized Feature Attribution for Tree Ensembles
• Iterative Random Forests to detect predictive and stable high-order interactions
• Signed iterative random forests to identify enhancer-associated transcription factor binding

Cited by in corpus (11)

• Unbiased variable importance for random forests
• Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance
• Trees, forests, and impurity-based variable importance
• Unbiased Measurement of Feature Importance in Tree-Based Methods
• Provable Boolean Interaction Recovery from Tree Ensemble obtained via Random Forests
• Business Model Contributions to Bank Profit Performance: A Machine Learning Approach
• Interpretable Anomaly Detection with DIFFI: Depth-based Isolation Forest Feature Importance
• Sharp Analysis of a Simple Model for Random Forests
• Bias, Fairness, and Accountability with AI and ML Algorithms
• From unbiased MDI Feature Importance to Explainable AI for Trees
• Data-driven advice for interpreting local and global model predictions in bioinformatics problems