Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

Abstract

Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: \url{https://github.com/thuml/Autoformer}.

References in corpus (5)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• WaveNet: A Generative Model for Raw Audio
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Reformer: The Efficient Transformer

Cited by in corpus (22)

• LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
• Generative Pretrained Hierarchical Transformer for Time Series Forecasting
• One for All: Unified Workload Prediction for Dynamic Multi-tenant Edge Cloud Platforms
• Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shift
• Monotonic Neural Ordinary Differential Equation: Time-series Forecasting for Cumulative Data
• OptScaler: A Collaborative Framework for Robust Autoscaling in the Cloud
• Emergency Vehicles Audio Detection and Localization in Autonomous Driving
• CrossLinear: Plug-and-Play Cross-Correlation Embedding for Time Series Forecasting with Exogenous Variables
• Unlocking the Potential of Deep Learning in Peak-Hour Series Forecasting
• SFTformer: A Spatial-Frequency-Temporal Correlation-Decoupling Transformer for Radar Echo Extrapolation
• Multiscale Representation Enhanced Temporal Flow Fusion Model for Long-Term Workload Forecasting
• Decoupling Spatio-Temporal Prediction: When Lightweight Large Models Meet Adaptive Hypergraphs
• From Known to Unknown: Knowledge-guided Transformer for Time-Series Sales Forecasting in Alibaba
• A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications
• ST-MTM: Masked Time Series Modeling with Seasonal-Trend Decomposition for Time Series Forecasting
• EnergyPatchTST: Multi-scale Time Series Transformers with Uncertainty Estimation for Energy Forecasting
• Noise-Resilient Point-wise Anomaly Detection in Time Series Using Weak Segment Labels
• Explainable and Interpretable Forecasts on Non-Smooth Multivariate Time Series for Responsible Gameplay
• S${}^3$Attention: Improving Long Sequence Attention with Smoothed Skeleton Sketching
• DeepClair: Utilizing Market Forecasts for Effective Portfolio Selection
• Before It's Too Late: A State Space Model for the Early Prediction of Misinformation and Disinformation Engagement
• D-Tracker: Modeling Interest Diffusion in Social Activity Tensor Data Streams