Multi-Layered Gradient Boosting Decision Trees

Abstract

Multi-layered representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are the dominant methods for modeling discrete or tabular data, they are hard to incorporate with such representation learning ability. In this work, we propose the multi-layered GBDT forest (mGBDTs), with an explicit emphasis on exploring the ability to learn hierarchical representations by stacking several layers of regression GBDTs as its building block. The model can be jointly trained by a variant of target propagation across layers, without the need to derive back-propagation nor differentiability. Experiments and visualizations confirmed the effectiveness of the model in terms of performance and representation learning ability.

Cited by in corpus (15)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
• Boost then Convolve: Gradient Boosting Meets Graph Neural Networks
• Privacy-Preserving Gradient Boosting Decision Trees
• Muddling Label Regularization: Deep Learning for Tabular Datasets
• Attention augmented differentiable forest for tabular data
• Learning Representations for Axis-Aligned Decision Forests through Input Perturbation
• A short note on the decision tree based neural turing machine
• Amended Cross Entropy Cost: Framework For Explicit Diversity Encouragement
• A Framework for an Assessment of the Kernel-target Alignment in Tree Ensemble Kernel Learning
• PACSET (Packed Serialized Trees): Reducing Inference Latency for Tree Ensemble Deployment
• Learning Multi-Layered GBDT Via Back Propagation
• Selective Cascade of Residual ExtraTrees
• Modeling Text with Decision Forests using Categorical-Set Splits