Gradient Boosting Neural Networks: GrowNet

Abstract

A novel gradient boosting framework is proposed where shallow neural networks are employed as ``weak learners''. General loss functions are considered under this unified framework with specific examples presented for classification, regression, and learning to rank. A fully corrective step is incorporated to remedy the pitfall of greedy function approximation of classic gradient boosting decision tree. The proposed model rendered outperforming results against state-of-the-art boosting methods in all three tasks on multiple datasets. An ablation study is performed to shed light on the effect of each model components and model hyperparameters.

Supplementary material starts after references

References in corpus (2)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks

Cited by in corpus (5)

• Revisiting Deep Learning Models for Tabular Data
• Boost then Convolve: Gradient Boosting Meets Graph Neural Networks
• A Gradient Boosting Approach for Training Convolutional and Deep Neural Networks
• Time-based Sequence Model for Personalization and Recommendation Systems
• Distributed Learning with Low Communication Cost via Gradient Boosting Untrained Neural Network