Self Attention with Temporal Prior: Can We Learn More from Arrow of Time?

Abstract

Many diverse phenomena in nature often inherently encode both short- and long-term temporal dependencies, which especially result from the direction of the flow of time. In this respect, we discovered experimental evidence suggesting that interrelations of these events are higher for closer time stamps. However, to be able for attention-based models to learn these regularities in short-term dependencies, it requires large amounts of data, which are often infeasible. This is because, while they are good at learning piece-wise temporal dependencies, attention-based models lack structures that encode biases in time series. As a resolution, we propose a simple and efficient method that enables attention layers to better encode the short-term temporal bias of these data sets by applying learnable, adaptive kernels directly to the attention matrices. We chose various prediction tasks for the experiments using Electronic Health Records (EHR) data sets since they are great examples with underlying long- and short-term temporal dependencies. Our experiments show exceptional classification results compared to best-performing models on most tasks and data sets.

References in corpus (7)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Multitask learning and benchmarking with clinical time series data
• Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
• Bidirectional Representation Learning from Transformers using Multimodal Electronic Health Record Data to Predict Depression
• Explicit Inductive Bias for Transfer Learning with Convolutional Networks
• Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier
• Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records