Deep Neural Networks and Tabular Data: A Survey

Abstract

Heterogeneous tabular data are the most commonly used form of data and are essential for numerous critical and computationally demanding applications. On homogeneous data sets, deep neural networks have repeatedly shown excellent performance and have therefore been widely adopted. However, their adaptation to tabular data for inference or data generation tasks remains challenging. To facilitate further progress in the field, this work provides an overview of state-of-the-art deep learning methods for tabular data. We categorize these methods into three groups: data transformations, specialized architectures, and regularization models. For each of these groups, our work offers a comprehensive overview of the main approaches. Moreover, we discuss deep learning approaches for generating tabular data, and we also provide an overview over strategies for explaining deep models on tabular data. Thus, our first contribution is to address the main research streams and existing methodologies in the mentioned areas, while highlighting relevant challenges and open research questions. Our second contribution is to provide an empirical comparison of traditional machine learning methods with eleven deep learning approaches across five popular real-world tabular data sets of different sizes and with different learning objectives. Our results, which we have made publicly available as competitive benchmarks, indicate that algorithms based on gradient-boosted tree ensembles still mostly outperform deep learning models on supervised learning tasks, suggesting that the research progress on competitive deep learning models for tabular data is stagnating. To the best of our knowledge, this is the first in-depth overview of deep learning approaches for tabular data; as such, this work can serve as a valuable starting point to guide researchers and practitioners interested in deep learning with tabular data.

References in corpus (42)

• Generative Adversarial Networks
• Transformers in Vision: A Survey
• Language Models are Few-Shot Learners
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
• Kaggle forecasting competitions: An overlooked learning opportunity
• Distilling a Neural Network Into a Soft Decision Tree
• The Cramer Distance as a Solution to Biased Wasserstein Gradients
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• Revisiting Deep Learning Models for Tabular Data
• Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems
• CTAB-GAN: Effective Table Data Synthesizing
• Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform
• On Embeddings for Numerical Features in Tabular Deep Learning
• An Open Source AutoML Benchmark
• Language Models are Realistic Tabular Data Generators
• Enhanced Security and Privacy via Fragmented Federated Learning
• ARM-Net: Adaptive Relation Modeling Network for Structured Data
• Copula Flows for Synthetic Data Generation
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• Network On Network for Tabular Data Classification in Real-world Applications
• Learning Parameter Distributions to Detect Concept Drift in Data Streams
• Boost then Convolve: Gradient Boosting Meets Graph Neural Networks
• MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask
• Synthetic Benchmarks for Scientific Research in Explainable Machine Learning
• On Baselines for Local Feature Attributions
• Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recourse
• CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms
• Leveraging Model Inherent Variable Importance for Stable Online Feature Selection
• Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Not All Datasets Are Born Equal: On Heterogeneous Data and Adversarial Examples
• Conservative Policy Construction Using Variational Autoencoders for Logged Data with Missing Values
• Causal-TGAN: Generating Tabular Data Using Causal Generative Adversarial Networks
• Sparse Sequence-to-Sequence Models
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• Simple Modifications to Improve Tabular Neural Networks
• Fairness in TabNet Model by Disentangled Representation for the Prediction of Hospital No-Show
• Muddling Label Regularization: Deep Learning for Tabular Datasets
• Synthesising Multi-Modal Minority Samples for Tabular Data
• Lockout: Sparse Regularization of Neural Networks

Cited by in corpus (48)

• Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review
• Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education
• Spectroscopic age estimates for APOGEE red-giant stars: Precise spatial and kinematic trends with age in the Galactic disc
• HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling
• Graph Neural Network contextual embedding for Deep Learning on Tabular Data
• Supervised Multiple Kernel Learning approaches for multi-omics data integration
• Using Machine Learning to Link Black Hole Accretion Flows with Spatially Resolved Polarimetric Observables
• Gradient Boosting Decision Trees on Medical Diagnosis over Tabular Data
• On the selection and effectiveness of pseudo-absences for species distribution modeling with deep learning
• Tabular Machine Learning Methods for Predicting Gas Turbine Emissions
• Diffusion Models for Tabular Data Imputation and Synthetic Data Generation
• Fuzzy Convolution Neural Networks for Tabular Data Classification
• Specification Overfitting in Artificial Intelligence
• Robust Semi-Supervised Learning in Open Environments
• FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
• NN2Poly: A polynomial representation for deep feed-forward artificial neural networks
• Energy reconstruction for large liquid scintillator detectors with machine learning techniques: aggregated features approach
• Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data
• Neural networks for insurance pricing with frequency and severity data: a benchmark study from data preprocessing to technical tariff
• A Federated Learning Benchmark on Tabular Data: Comparing Tree-Based Models and Neural Networks
• Endogenous Macrodynamics in Algorithmic Recourse
• A generalized decision tree ensemble based on the NeuralNetworks architecture: Distributed Gradient Boosting Forest (DGBF)
• Diffusion-Scheduled Denoising Autoencoders for Anomaly Detection in Tabular Data
• Robust-Multi-Task Gradient Boosting
• Artificial Inductive Bias for Synthetic Tabular Data Generation in Data-Scarce Scenarios
• Towards Cardiac MRI Foundation Models: Comprehensive Visual-Tabular Representations for Whole-Heart Assessment and Beyond
• Privacy Constrained Fairness Estimation for Decision Trees
• Predictive Model Development to Identify Failed Healing in Patients after Non-Union Fracture Surgery
• Comparison of generalised additive models and neural networks in applications: A systematic review
• TabAttackBench: A Benchmark for Adversarial Attacks on Tabular Data
• Histogram-Based Federated XGBoost using Minimal Variance Sampling for Federated Tabular Data
• Trustworthy Representation Learning via Information Funnels and Bottlenecks
• RO-FIGS: Efficient and Expressive Tree-Based Ensembles for Tabular Data
• Tabular Image: a method to convert tabular data to images for convolutional neural networks
• T-Explainer: A Model-Agnostic Explainability Framework Based on Gradients
• GraphFire-X: Physics-Informed Graph Attention Networks and Structural Gradient Boosting for Building-Scale Wildfire Preparedness at the Wildland-Urban Interface
• Investigating Group Distributionally Robust Optimization for Deep Imbalanced Learning: A Case Study of Binary Tabular Data Classification
• Fast and Accurate Explanations of Distance-Based Classifiers by Uncovering Latent Explanatory Structures
• Learning Pole Structures of Hadronic States using Predictive Uncertainty Estimation
• Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunities
• A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines
• Machine Learning in Epidemiology
• Improving ASP-based ORS Schedules through Machine Learning Predictions
• Crafting Imperceptible On-Manifold Adversarial Attacks for Tabular Data
• Backdoor attacks on DNN and GBDT -- A Case Study from the insurance domain
• ML4EJ: Decoding the Role of Urban Features in Shaping Environmental Injustice Using Interpretable Machine Learning
• A Method for Discovering Novel Classes in Tabular Data
• Sparse tree-based initialization for neural networks