Transformers predicting the future. Applying attention in next-frame and time series forecasting

Abstract

Recurrent Neural Networks were, until recently, one of the best ways to capture the timely dependencies in sequences. However, with the introduction of the Transformer, it has been proven that an architecture with only attention-mechanisms without any RNN can improve on the results in various sequence processing tasks (e.g. NLP). Multiple studies since then have shown that similar approaches can be applied for images, point clouds, video, audio or time series forecasting. Furthermore, solutions such as the Perceiver or the Informer have been introduced to expand on the applicability of the Transformer. Our main objective is testing and evaluating the effectiveness of applying Transformer-like models on time series data, tackling susceptibility to anomalies, context awareness and space complexity by fine-tuning the hyperparameters, preprocessing the data, applying dimensionality reduction or convolutional encodings, etc. We are also looking at the problem of next-frame prediction and exploring ways to modify existing solutions in order to achieve higher performance and learn generalized knowledge.

arXiv comment8 pages, 5 figures. Written during Summer Research School 2021 in Apriltsi, Bulgaria

References in corpus (11)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Neural Discrete Representation Learning
• Unsupervised Learning of Video Representations using LSTMs
• Generating Long Sequences with Sparse Transformers
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Axial Attention in Multidimensional Transformers
• CvT: Introducing Convolutions to Vision Transformers
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Perceiver: General Perception with Iterative Attention
• ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis
• Open-Ended Learning Leads to Generally Capable Agents