Deep differentiable forest with sparse attention for the tabular data
arXiv:2003.00223
Abstract
We present a general architecture of deep differentiable forest and its sparse attention mechanism. The differentiable forest has the advantages of both trees and neural networks. Its structure is a simple binary tree, easy to use and understand. It has full differentiability and all variables are learnable parameters. We would train it by the gradient-based optimization method, which shows great power in the training of deep CNN. We find and analyze the attention mechanism in the differentiable forest. That is, each decision depends on only a few important features, and others are irrelevant. The attention is always sparse. Based on this observation, we improve its sparsity by data-aware initialization. We use the attribute importance to initialize the attention weight. Then the learned weight is much sparse than that from random initialization. Our experiment on some large tabular dataset shows differentiable forest has higher accuracy than GBDT, which is the state of art algorithm for tabular datasets. The source codes are available at https://github.com/closest-git/QuantumForest
6 pages,3 figures
References in corpus (9)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- CatBoost: gradient boosting with categorical features support
- Searching for Exotic Particles in High-Energy Physics with Deep Learning
- Introducing LETOR 4.0 Datasets
- Deep Neural Decision Trees
- Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
- Optimization Methods for Interpretable Differentiable Decision Trees in Reinforcement Learning
- Random Hinge Forest for Differentiable Learning
- LiteMORT: A memory efficient gradient boosting tree system on adaptive compact distributions