Long-Range Transformers for Dynamic Spatiotemporal Forecasting
arXiv:2109.12218
Abstract
Multivariate time series forecasting focuses on predicting future values based on historical context. State-of-the-art sequence-to-sequence models rely on neural attention between timesteps, which allows for temporal learning but fails to consider distinct spatial relationships between variables. In contrast, methods based on graph neural networks explicitly model variable relationships. However, these methods often rely on predefined graphs that cannot change over time and perform separate spatial and temporal updates without establishing direct connections between each variable at every timestep. Our work addresses these problems by translating multivariate forecasting into a "spatiotemporal sequence" formulation where each Transformer input token represents the value of a single variable at a given time. Long-Range Transformers can then learn interactions between space, time, and value information jointly along this extended sequence. Our method, which we call Spacetimeformer, achieves competitive results on benchmarks from traffic forecasting to electricity demand and weather prediction while learning spatiotemporal relationships purely from data.
References in corpus (25)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Spectral Networks and Locally Connected Networks on Graphs
- Time Series Forecasting With Deep Learning: A Survey
- Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
- Linformer: Self-Attention with Linear Complexity
- Generating Long Sequences with Sparse Transformers
- Evaluating time series forecasting models: An empirical study on performance estimation methods
- Conditional Time Series Forecasting with Convolutional Neural Networks
- A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
- Reducing Transformer Depth on Demand with Structured Dropout
- Spatial-Temporal Transformer Networks for Traffic Flow Forecasting
- Big Bird: Transformers for Longer Sequences
- Long Range Arena: A Benchmark for Efficient Transformers
- ST-GRAT: A Novel Spatio-temporal Graph Attention Network for Accurately Forecasting Dynamically Changing Road Speed
- Rethinking Attention with Performers
- Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks
- Long-Short Transformer: Efficient Transformers for Language and Vision
- BP-Transformer: Modelling Long-Range Context via Binary Partitioning
- Time2Vec: Learning a Vector Representation of Time
- Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
- Understanding the Difficulty of Training Transformers
- PowerNorm: Rethinking Batch Normalization in Transformers
- Few-shot Learning for Time-series Forecasting
- Scheduled DropHead: A Regularization Method for Transformer Models
- UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost