Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance

Abstract

As the size, complexity, and availability of data continues to grow, scientists are increasingly relying upon black-box learning algorithms that can often provide accurate predictions with minimal a priori model specifications. Tools like random forests have an established track record of off-the-shelf success and even offer various strategies for analyzing the underlying relationships among variables. Here, motivated by recent insights into random forest behavior, we introduce the simple idea of augmented bagging (AugBagg), a procedure that operates in an identical fashion to classical bagging and random forests, but which operates on a larger, augmented space containing additional randomly generated noise features. Surprisingly, we demonstrate that this simple act of including extra noise variables in the model can lead to dramatic improvements in out-of-sample predictive accuracy, sometimes outperforming even an optimally tuned traditional random forest. As a result, intuitive notions of variable importance based on improved model accuracy may be deeply flawed, as even purely random noise can routinely register as statistically significant. Numerous demonstrations on both real and synthetic data are provided along with a proposed solution.

References in corpus (10)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Consistency of random forests
• Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso
• Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Scalable and Efficient Hypothesis Testing with Random Forests
• The Implicit Regularization of Ordinary Least Squares Ensembles
• Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics
• Sharp Analysis of a Simple Model for Random Forests
• Implicit Regularization of Random Feature Models

Cited by in corpus (4)

• Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance
• Sequential Permutation Testing of Random Forest Variable Importance Measures
• Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics
• Random Forests for dependent data