Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success
arXiv:1911.00190
Abstract
Random forests remain among the most popular off-the-shelf supervised machine learning tools with a well-established track record of predictive accuracy in both regression and classification settings. Despite their empirical success as well as a bevy of recent work investigating their statistical properties, a full and satisfying explanation for their success has yet to be put forth. Here we aim to take a step forward in this direction by demonstrating that the additional randomness injected into individual trees serves as a form of implicit regularization, making random forests an ideal model in low signal-to-noise ratio (SNR) settings. Specifically, from a model-complexity perspective, we show that the mtry parameter in random forests serves much the same purpose as the shrinkage penalty in explicitly regularized regression procedures like lasso and ridge regression. To highlight this point, we design a randomized linear-model-based forward selection procedure intended as an analogue to tree-based random forests and demonstrate its surprisingly strong empirical performance. Numerous demonstrations on both real and synthetic data are provided.
arXiv commentTo Appear in the Journal of Machine Learning Research (JMLR)
References in corpus (16)
- Reconciling modern machine learning practice and the bias-variance trade-off
- OpenML: networked science in machine learning
- Analysis of a Random Forests Model
- Consistency of random forests
- Dropout Training as Adaptive Regularization
- Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso
- Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests
- Random Forests: some methodological insights
- To tune or not to tune the number of trees in random forest?
- Making Sense of Random Forest Probabilities: a Kernel Perspective
- Scalable and Efficient Hypothesis Testing with Random Forests
- Consistency of survival tree and forest models: splitting bias and correction
- The Implicit Regularization of Ordinary Least Squares Ensembles
- Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics
- Sharp Analysis of a Simple Model for Random Forests
- Impact of subsampling and pruning on random forests