Label Attention Network for Temporal Sets Prediction: You Were Looking at a Wrong Self-Attention

Abstract

Most user-related data can be represented as a sequence of events associated with a timestamp and a collection of categorical labels. For example, the purchased basket of goods and the time of buying fully characterize the event of the store visit. Anticipation of the label set for the future event called the problem of temporal sets prediction, holds significant value, especially in such high-stakes industries as finance and e-commerce. A fundamental challenge of this task is the joint consideration of the temporal nature of events and label relations within sets. The existing models fail to capture complex time and label dependencies due to ineffective representation of historical information initially. We aim to address this shortcoming by presenting the framework with a specific way to aggregate the observed information into time- and set structure-aware views prior to transferring it into main architecture blocks. Our strong emphasis on input arrangement facilitates the subsequent efficient learning of label interactions. The proposed model is called Label-Attention NETwork, or LANET. We conducted experiments on four different datasets and made a comparison with four established models, including SOTA, in this area. The experimental results suggest that LANET provides significantly better quality than any other model, achieving an improvement up to $65\%$ in terms of weighted F1 metric compared to the closest competitor. Moreover, we contemplate causal relationships between labels in our work, as well as a thorough study of LANET components' influence on performance. We provide an implementation of LANET to encourage its wider usage.

References in corpus (9)

• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• ML-Net: multi-label classification of biomedical texts with deep neural networks
• Gated Transformer Networks for Multivariate Time Series Classification
• CoLES: Contrastive Learning for Event Sequences with Self-Supervision
• Transformer Embeddings of Irregularly Spaced Events and Their Participants
• Multi-label Prediction in Time Series Data using Deep Neural Networks
• Learning Transactions Representations for Information Management in Banks: Mastering Local, Global, and External Knowledge
• Continuous-time convolutions model of event sequences
• Causal Interpretation of Self-Attention in Pre-Trained Transformers