Revisiting Deep Learning Models for Tabular Data

Abstract

The existing literature on deep learning for tabular data proposes a wide range of novel architectures and reports competitive results on various datasets. However, the proposed models are usually not properly compared to each other and existing works often use different benchmarks and experiment protocols. As a result, it is unclear for both researchers and practitioners what models perform best. Additionally, the field still lacks effective baselines, that is, the easy-to-use models that provide competitive performance across different problems. In this work, we perform an overview of the main families of DL architectures for tabular data and raise the bar of baselines in tabular DL by identifying two simple and powerful deep architectures. The first one is a ResNet-like architecture which turns out to be a strong baseline that is often missing in prior works. The second model is our simple adaptation of the Transformer architecture for tabular data, which outperforms other solutions on most tasks. Both models are compared to many existing architectures on a diverse set of tasks under the same training and tuning protocols. We also compare the best DL models with Gradient Boosted Decision Trees and conclude that there is still no universally superior solution.

NeurIPS 2021 camera-ready. Code: https://github.com/yandex-research/tabular-dl-revisiting-models (v3-v5: minor changes)

References in corpus (8)

• OpenML: networked science in machine learning
• Linformer: Self-Attention with Linear Complexity
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings
• Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
• Gradient Boosting Neural Networks: GrowNet
• GLU Variants Improve Transformer
• The Tree Ensemble Layer: Differentiability meets Conditional Computation
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention

Cited by in corpus (9)

• Deep Neural Networks and Tabular Data: A Survey
• SurvTRACE: Transformers for Survival Analysis with Competing Events
• Anomaly Detection with Score Distribution Discrimination
• Comparing Algorithm Selection Approaches on Black-Box Optimization Problems
• FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
• TABCF: Counterfactual Explanations for Tabular Data Using a Transformer-Based VAE
• Simple Modifications to Improve Tabular Neural Networks
• Machine Learning Approaches to Automated Flow Cytometry Diagnosis of Chronic Lymphocytic Leukemia
• Look behind the Censorship: Reposting-User Characterization and Muted-Topic Restoration