Scalable and Efficient Hypothesis Testing with Random Forests

Abstract

Throughout the last decade, random forests have established themselves as among the most accurate and popular supervised learning methods. While their black-box nature has made their mathematical analysis difficult, recent work has established important statistical properties like consistency and asymptotic normality by considering subsampling in lieu of bootstrapping. Though such results open the door to traditional inference procedures, all formal methods suggested thus far place severe restrictions on the testing framework and their computational overhead precludes their practical scientific use. Here we propose a permutation-style testing approach to formally assess feature significance. We establish asymptotic validity of the test via exchangeability arguments and show that the test maintains high power with orders of magnitude fewer computations. As importantly, the procedure scales easily to big data settings where large training and testing sets may be employed without the need to construct additional models. Simulations and applications to ecological data where random forests have recently shown promise are provided.

arXiv comment52 pages, 10 figures [fixed some critical typo's with Algorithm 1]

References in corpus (6)

• Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn
• Analysis of a Random Forests Model
• Consistency of random forests
• Exact and asymptotically robust permutation tests
• Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests
• Consistency of survival tree and forest models: splitting bias and correction

Cited by in corpus (7)

• Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance
• Sequential Permutation Testing of Random Forest Variable Importance Measures
• Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success
• Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance
• Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics
• Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning
• Locally Optimized Random Forests