Uncertainty in Gradient Boosting via Ensembles

Abstract

For many practical, high-risk applications, it is essential to quantify uncertainty in a model's predictions to avoid costly mistakes. While predictive uncertainty is widely studied for neural networks, the topic seems to be under-explored for models based on gradient boosting. However, gradient boosting often achieves state-of-the-art results on tabular data. This work examines a probabilistic ensemble-based framework for deriving uncertainty estimates in the predictions of gradient boosting classification and regression models. We conducted experiments on a range of synthetic and real datasets and investigated the applicability of ensemble approaches to gradient boosting models that are themselves ensembles of decision trees. Our analysis shows that ensembles of gradient boosting models successfully detect anomalous inputs while having limited ability to improve the predicted total uncertainty. Importantly, we also propose a concept of a virtual ensemble to get the benefits of an ensemble via only one gradient boosting model, which significantly reduces complexity.

References in corpus (10)

• Scikit-learn: Machine Learning in Python
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Boosted Decision Trees as an Alternative to Artificial Neural Networks for Particle Identification
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness
• SGLB: Stochastic Gradient Langevin Boosting

Cited by in corpus (4)

• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Advancing Molecular Machine Learning Representations with Stereoelectronics-Infused Molecular Graphs
• SeqNAS: Neural Architecture Search for Event Sequence Classification
• TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression