A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Abstract

We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.

Accepted by ICLR 2023

Cited by in corpus (17)

• TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting
• LSTTN: A Long-Short Term Transformer-based Spatio-temporal Neural Network for Traffic Flow Forecasting
• Real-Time Decision-Making for Digital Twin in Additive Manufacturing with Model Predictive Control using Time-Series Deep Neural Networks
• How Does It Function? Characterizing Long-term Trends in Production Serverless Workloads
• Joint Estimation and Prediction of City-wide Delivery Demand: A Large Language Model Empowered Graph-based Learning Approach
• Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations
• Astroconformer: The Prospects of Analyzing Stellar Light Curves with Transformer-Based Deep Learning Models
• A General Framework for Generative Self-supervised Learning in Non-invasive Estimation of Physiological Parameters Using Photoplethysmography
• ExoTST: Exogenous-Aware Temporal Sequence Transformer for Time Series Prediction
• EnergyPatchTST: Multi-scale Time Series Transformers with Uncertainty Estimation for Energy Forecasting
• Reducing Label Dependency in Human Activity Recognition with Wearables: From Supervised Learning to Novel Weakly Self-Supervised Approaches
• Koopman Learning with Episodic Memory
• Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations
• Domain Adaptation for Time series Transformers using One-step fine-tuning
• Multi-grained spatial-temporal feature complementarity for accurate online cellular traffic prediction
• Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecasting
• sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging