Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data

Abstract

Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of heterogenous tabular data, the advantage of DNNs over shallow counterparts remains questionable. In particular, there is no sufficient evidence that deep learning machinery allows constructing methods that outperform gradient boosting decision trees (GBDT), which are often the top choice for tabular problems. In this paper, we introduce Neural Oblivious Decision Ensembles (NODE), a new deep learning architecture, designed to work with any tabular data. In a nutshell, the proposed NODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning. With an extensive experimental comparison to the leading GBDT packages on a large number of tabular datasets, we demonstrate the advantage of the proposed NODE architecture, which outperforms the competitors on most of the tasks. We open-source the PyTorch implementation of NODE and believe that it will become a universal framework for machine learning on tabular data.

References in corpus (10)

• Categorical Reparameterization with Gumbel-Softmax
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Averaging Weights Leads to Wider Optima and Better Generalization
• Deep Neural Decision Trees
• Multi-Layered Gradient Boosting Decision Trees
• Quasi-hyperbolic momentum and Adam for deep learning
• A Regularized Framework for Sparse and Structured Neural Attention
• SparseMAP: Differentiable Sparse Structured Inference
• Forward Thinking: Building Deep Random Forests
• Random Hinge Forest for Differentiable Learning

Cited by in corpus (16)

• Deep Neural Networks and Tabular Data: A Survey
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Learning Global Transparent Models Consistent with Local Contrastive Explanations
• Deep differentiable forest with sparse attention for the tabular data
• Controlling Neural Networks with Rule Representations
• PyTorch Tabular: A Framework for Deep Learning with Tabular Data
• Learning Unsplit-field-based PML for the FDTD Method by Deep Differentiable Forest
• Simple Modifications to Improve Tabular Neural Networks
• Tab-Attention: Self-Attention-based Stacked Generalization for Imbalanced Credit Default Prediction
• Attention augmented differentiable forest for tabular data
• Efficient Encrypted Inference on Ensembles of Decision Trees
• Dive into Decision Trees and Forests: A Theoretical Demonstration
• Yet Another Representation of Binary Decision Trees: A Mathematical Demonstration
• A short note on the decision tree based neural turing machine