SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

Abstract

Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even outperforms gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over a variety of benchmark tasks.

References in corpus (12)

• mixup: Beyond Empirical Risk Minimization
• Bootstrap your own latent: A new approach to self-supervised Learning
• Gaussian Error Linear Units (GELUs)
• CatBoost: gradient boosting with categorical features support
• Generating Long Sequences with Sparse Transformers
• Exploring Simple Siamese Representation Learning
• Axial Attention in Multidimensional Transformers
• Fifty Years of Pulsar Candidate Selection: From simple filters to a new principled real-time classification approach
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• Regularization Learning Networks: Deep Learning for Tabular Datasets
• Learning Semantic Annotations for Tabular Data

Cited by in corpus (8)

• Deep Neural Networks and Tabular Data: A Survey
• Graph Neural Network contextual embedding for Deep Learning on Tabular Data
• Learning Interpretable Rules for Scalable Data Representation and Classification
• FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data
• Local Contrastive Feature learning for Tabular Data
• Beyond Individual Input for Deep Anomaly Detection on Tabular Data
• CpT: Convolutional Point Transformer for 3D Point Cloud Processing
• Sparse tree-based initialization for neural networks