Variable importance in binary regression trees and forests

Abstract

We characterize and study variable importance (VIMP) and pairwise variable associations in binary regression trees. A key component involves the node mean squared error for a quantity we refer to as a maximal subtree. The theory naturally extends from single trees to ensembles of trees and applies to methods like random forests. This is useful because while importance values from random forests are used to screen variables, for example they are used to filter high throughput genomic data in Bioinformatics, very little theory exists about their properties.

Published in at http://dx.doi.org/10.1214/07-EJS039 the Electronic Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Cited by in corpus (12)

• Random survival forests
• Correlation and variable importance in random forests
• Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
• An Interpretable Probabilistic Model for Short-Term Solar Power Forecasting Using Natural Gradient Boosting
• A general framework for inference on algorithm-agnostic variable importance
• Hierarchical Clustering for Smart Meter Electricity Loads based on Quantile Autocovariances
• Boosting Random Forests to Reduce Bias; One-Step Boosted Forest and its Variance Estimate
• Formal Verification of Decision-Tree Ensemble Model and Detection of its Violating-input-value Ranges
• Unboxing Tree Ensembles for interpretability: a hierarchical visualization tool and a multivariate optimal re-built tree
• Provable Boolean Interaction Recovery from Tree Ensemble obtained via Random Forests
• Machine learning approaches to identify thresholds in a heat-health warning system context
• Variable Selection with ABC Bayesian Forests