Transformers in Time-series Analysis: A Tutorial

Abstract

Transformer architecture has widespread applications, particularly in Natural Language Processing and computer vision. Recently Transformers have been employed in various aspects of time-series analysis. This tutorial provides an overview of the Transformer architecture, its applications, and a collection of examples from recent research papers in time-series analysis. We delve into an explanation of the core components of the Transformer, including the self-attention mechanism, positional encoding, multi-head, and encoder/decoder. Several enhancements to the initial, Transformer architecture are highlighted to tackle time-series tasks. The tutorial also provides best practices and techniques to overcome the challenge of effectively training Transformers for time-series analysis.

28 pages, 17 figures

References in corpus (4)

• Decision Transformer: Reinforcement Learning via Sequence Modeling
• TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• Exploring Robust Architectures for Deep Artificial Neural Networks

Cited by in corpus (5)

• Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review
• LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
• Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detection
• Domain Adaptation for Time series Transformers using One-step fine-tuning
• Integrating the Expected Future in Load Forecasts with Contextually Enhanced Transformer Models