Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution
arXiv:1806.09762
Abstract
This paper examines a novel gradient boosting framework for regression. We regularize gradient boosted trees by introducing subsampling and employ a modified shrinkage algorithm so that at every boosting stage the estimate is given by an average of trees. The resulting algorithm, titled Boulevard, is shown to converge as the number of trees grows. We also demonstrate a central limit theorem for this limit, allowing a characterization of uncertainty for predictions. A simulation study and real world examples provide support for both the predictive accuracy of the model and its limiting behavior.
45 pages, 7 figures
References in corpus (9)
- Analysis of a Random Forests Model
- Dropout Training as Adaptive Regularization
- Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests
- DART: Dropouts meet Multiple Additive Regression Trees
- Adaptive Concentration of Regression Trees, with Application to Random Forests
- Analysis of purely random forests bias
- The Random Forest Kernel and other kernels for big data from random partitions
- Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics
- InfiniteBoost: building infinite ensembles with gradient descent