TabTransformer: Tabular Data Modeling Using Contextual Embeddings

Abstract

We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. Through extensive experiments on fifteen publicly available datasets, we show that the TabTransformer outperforms the state-of-the-art deep learning methods for tabular data by at least 1.0% on mean AUC, and matches the performance of tree-based ensemble models. Furthermore, we demonstrate that the contextual embeddings learned from TabTransformer are highly robust against both missing and noisy data features, and provide better interpretability. Lastly, for the semi-supervised setting we develop an unsupervised pre-training procedure to learn data-driven contextual embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art methods.

7 pages, 5 figures

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• word2vec Parameter Learning Explained
• Visualizing and Measuring the Geometry of BERT
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Deep Neural Decision Trees
• On Identifiability in Transformers
• Uncertainty in the Variational Information Bottleneck
• DeepFM: An End-to-End Wide & Deep Learning Framework for CTR Prediction
• Semi-Supervised Learning with Normalizing Flows

Cited by in corpus (18)

• Deep Neural Networks and Tabular Data: A Survey
• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• Revisiting Deep Learning Models for Tabular Data
• Graph Neural Network contextual embedding for Deep Learning on Tabular Data
• MARIA: a Multimodal Transformer Model for Incomplete Healthcare Data
• Enhancing Actuarial Non-Life Pricing Models via Transformers
• Automated Essay Scoring Using Transformer Models
• A Federated Learning Benchmark on Tabular Data: Comparing Tree-Based Models and Neural Networks
• Association in Facial Phenotype, Gene, Disease: A Dataset for Explainable Rare Genetic Diseases Diagnosis
• Insights into Dark Matter Direct Detection Experiments: Decision Trees versus Deep Learning
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields
• Simple Modifications to Improve Tabular Neural Networks
• Explainable AI for Infection Prevention and Control: Modeling CPE Acquisition and Patient Outcomes in an Irish Hospital with Transformers
• A Novel Unified Lightweight Temporal-Spatial Transformer Approach for Intrusion Detection in Drone Networks
• Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features
• Bond Default Prediction with Text Embeddings, Undersampling and Deep Learning