W-Transformers : A Wavelet-based Transformer Framework for Univariate Time Series Forecasting

Abstract

Deep learning utilizing transformers has recently achieved a lot of success in many vital areas such as natural language processing, computer vision, anomaly detection, and recommendation systems, among many others. Among several merits of transformers, the ability to capture long-range temporal dependencies and interactions is desirable for time series forecasting, leading to its progress in various time series applications. In this paper, we build a transformer model for non-stationary time series. The problem is challenging yet crucially important. We present a novel framework for univariate time series representation learning based on the wavelet-based transformer encoder architecture and call it W-Transformer. The proposed W-Transformers utilize a maximal overlap discrete wavelet transformation (MODWT) to the time series data and build local transformers on the decomposed datasets to vividly capture the nonstationarity and long-range nonlinear dependencies in the time series. Evaluating our framework on several publicly available benchmark time series datasets from various domains and with diverse characteristics, we demonstrate that it performs, on average, significantly better than the baseline forecasters for short-term and long-term forecasting, even for datasets that consist of only a few hundred training samples.

References in corpus (7)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Time Series Forecasting With Deep Learning: A Survey
• Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions
• Real-time forecasts and risk assessment of novel coronavirus (COVID-19) cases: A data-driven analysis
• Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
• Epicasting: An Ensemble Wavelet Neural Network (EWNet) for Forecasting Epidemics

Cited by in corpus (6)

• Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art
• When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challenges
• Epicasting: An Ensemble Wavelet Neural Network (EWNet) for Forecasting Epidemics
• WaveCatBoost for Probabilistic Forecasting of Regional Air Quality Data
• Epidemic-guided deep learning for spatiotemporal forecasting of Tuberculosis outbreak
• Spatiotemporal Forecasting of Traffic Flow using Wavelet-based Temporal Attention