Scalable Transformer for High Dimensional Multivariate Time Series Forecasting

Abstract

Deep models for Multivariate Time Series (MTS) forecasting have recently demonstrated significant success. Channel-dependent models capture complex dependencies that channel-independent models cannot capture. However, the number of channels in real-world applications outpaces the capabilities of existing channel-dependent models, and contrary to common expectations, some models underperform the channel-independent models in handling high-dimensional data, which raises questions about the performance of channel-dependent models. To address this, our study first investigates the reasons behind the suboptimal performance of these channel-dependent models on high-dimensional MTS data. Our analysis reveals that two primary issues lie in the introduced noise from unrelated series that increases the difficulty of capturing the crucial inter-channel dependencies, and challenges in training strategies due to high-dimensional data. To address these issues, we propose STHD, the Scalable Transformer for High-Dimensional Multivariate Time Series Forecasting. STHD has three components: a) Relation Matrix Sparsity that limits the noise introduced and alleviates the memory issue; b) ReIndex applied as a training strategy to enable a more flexible batch size setting and increase the diversity of training data; and c) Transformer that handles 2-D inputs and captures channel dependencies. These components jointly enable STHD to manage the high-dimensional MTS while maintaining computational feasibility. Furthermore, experimental results show STHD's considerable improvement on three high-dimensional datasets: Crime-Chicago, Wiki-People, and Traffic. The source code and dataset are publicly available https://github.com/xinzzzhou/ScalableTransformer4HighDimensionMTSF.git.

References in corpus (13)

• iTransformer: Inverted Transformers Are Effective for Time Series Forecasting
• Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting
• TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting
• TFAD: A Decomposition Time Series Anomaly Detection Architecture with Time-Frequency Analysis
• Temporal and Heterogeneous Graph Neural Network for Financial Time Series Prediction
• Domain Adversarial Spatial-Temporal Network: A Transferable Framework for Short-term Traffic Forecasting across Cities
• Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Series
• Adaptive Dependency Learning Graph Neural Networks
• Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders
• DuETA: Traffic Congestion Propagation Pattern Modeling via Efficient Graph Learning for ETA Prediction at Baidu Maps
• DeepSTA: A Spatial-Temporal Attention Network for Logistics Delivery Timely Rate Prediction in Anomaly Conditions
• MemDA: Forecasting Urban Time Series with Memory-based Drift Adaptation
• STGIN: Spatial-Temporal Graph Interaction Network for Large-scale POI Recommendation