Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests

Abstract

This work develops formal statistical inference procedures for machine learning ensemble methods. Ensemble methods based on bootstrapping, such as bagging and random forests, have improved the predictive accuracy of individual trees, but fail to provide a framework in which distributional results can be easily determined. Instead of aggregating full bootstrap samples, we consider predicting by averaging over trees built on subsamples of the training set and demonstrate that the resulting estimator takes the form of a U-statistic. As such, predictions for individual feature vectors are asymptotically normal, allowing for confidence intervals to accompany predictions. In practice, a subset of subsamples is used for computational speed; here our estimators take the form of incomplete U-statistics and equivalent results are derived. We further demonstrate that this setup provides a framework for testing the significance of features. Moreover, the internal estimation method we develop allows us to estimate the variance parameters and perform these inference procedures at no additional computational cost. Simulations and illustrations on a real dataset are provided.

To appear in The Journal of Machine Learning Research

References in corpus (3)

• Narrowing the Gap: Random Forests In Theory and In Practice
• Variable selection for BART: An application to gene regulation
• Asymptotic Theory for Random Forests

Cited by in corpus (40)

• All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously
• Data-driven polynomial chaos expansion for machine learning regression
• Data-Driven Modelling of the Reynolds Stress Tensor using Random Forests with Invariance
• Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry
• Boosting Random Forests to Reduce Bias; One-Step Boosted Forest and its Variance Estimate
• Interpretable Random Forests via Rule Extraction
• Approximating high-dimensional infinite-order $U$-statistics: statistical and computational guarantees
• Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance
• Predictive Inference Is Free with the Jackknife+-after-Bootstrap
• Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena
• Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success
• Scalable and Efficient Hypothesis Testing with Random Forests
• Estimation and Inference with Trees and Forests in High Dimensions
• Unbiased Measurement of Feature Importance in Tree-Based Methods
• SHAFF: Fast and consistent SHApley eFfect estimates via random Forests
• Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance
• Provable Boolean Interaction Recovery from Tree Ensemble obtained via Random Forests
• Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics
• Sharp Analysis of a Simple Model for Random Forests
• A Unified Framework for Random Forest Prediction Error Estimation
• Impact of subsampling and pruning on random forests
• Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution
• Extrapolated cross-validation for randomized ensembles
• $V$-statistics and Variance Estimation
• Random Forests for dependent data
• Randomized incomplete $U$-statistics in high dimensions
• Bias, Consistency, and Alternative Perspectives of the Infinitesimal Jackknife
• Tree Boosted Varying Coefficient Models
• On the Subbagging Estimation for Massive Data
• Random Planted Forest: a directly interpretable tree ensemble
• On the Consistency of a Random Forest Algorithm in the Presence of Missing Entries
• Forest Guided Smoothing
• Estimating the Algorithmic Variance of Randomized Ensembles via the Bootstrap
• Locally Optimized Random Forests
• Modelling hetegeneous treatment effects by quantitle local polynomial decision tree and forest
• Towards Robust Classification with Deep Generative Forests
• Asymptotic Unbiasedness of the Permutation Importance Measure in Random Forest Models
• Unaware Fairness: Hierarchical Random Forest for Protected Classes
• EVIboost for the Estimation of Extreme Value Index under Heterogeneous Extremes
• Generalised Boosted Forests