FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data

Abstract

Sequential tabular data is one of the most commonly used data types in real-world applications. Different from conventional tabular data, where rows in a table are independent, sequential tabular data contains rich contextual and sequential information, where some fields are dynamically changing over time and others are static. Existing transformer-based approaches analyzing sequential tabular data overlook the differences between dynamic and static fields by replicating and filling static fields into each transformer, and ignore temporal information between rows, which leads to three major disadvantages: (1) computational overhead, (2) artificially simplified data for masked language modeling pre-training task that may yield less meaningful representations, and (3) disregarding the temporal behavioral patterns implied by time intervals. In this work, we propose FATA-Trans, a model with two field transformers for modeling sequential tabular data, where each processes static and dynamic field information separately. FATA-Trans is field- and time-aware for sequential tabular data. The field-type embedding in the method enables FATA-Trans to capture differences between static and dynamic fields. The time-aware position embedding exploits both order and time interval information between rows, which helps the model detect underlying temporal behavior in a sequence. Our experiments on three benchmark datasets demonstrate that the learned representations from FATA-Trans consistently outperform state-of-the-art solutions in the downstream tasks. We also present visualization studies to highlight the insights captured by the learned representations, enhancing our understanding of the underlying data. Our codes are available at https://github.com/zdy93/FATA-Trans.

arXiv commentThis work is accepted by ACM International Conference on Information and Knowledge Management (CIKM) 2023

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Deep Learning based Recommender System: A Survey and New Perspectives
• Deep Neural Networks and Tabular Data: A Survey
• AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks
• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• Revisiting Deep Learning Models for Tabular Data
• Denoising Self-attentive Sequential Recommendation
• Interleaved Sequence RNNs for Fraud Detection
• On Embeddings for Numerical Features in Tabular Deep Learning
• Transfer Learning with Deep Tabular Models
• Attributed Sequence Embedding
• Revisiting Pretraining Objectives for Tabular Deep Learning
• One Transformer for All Time Series: Representing and Training with Time-Dependent Heterogeneous Tabular Data
• LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training