Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

Abstract

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length $L$, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L{)}^2)$ memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.

To appear in the proceeding of NeurIPS 2019

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• On the difficulty of training Recurrent Neural Networks
• WaveNet: A Generative Model for Raw Audio
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Generating Long Sequences with Sparse Transformers
• Conditional Time Series Forecasting with Convolutional Neural Networks
• A Multi-Horizon Quantile Recurrent Forecaster
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Generating Wikipedia by Summarizing Long Sequences
• Long-term Forecasting using Higher Order Tensor RNNs
• Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
• Recurrent Batch Normalization
• Effective Bayesian Modeling of Groups of Related Count Time Series
• Deep Factors with Gaussian Processes for Forecasting

Cited by in corpus (55)

• Time Series Forecasting With Deep Learning: A Survey
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Deep Learning for Time Series Forecasting: Tutorial and Literature Survey
• SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
• Spatio-Temporal Wind Speed Forecasting using Graph Networks and Novel Transformer Architectures
• Transformers in Healthcare: A Survey
• TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting
• Long-Range Transformers for Dynamic Spatiotemporal Forecasting
• Unsupervised Deep Learning for IoT Time Series
• Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows
• Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
• Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression
• Detecting Intentional AIS Shutdown in Open Sea Maritime Surveillance Using Self-Supervised Deep Learning
• Adaptive Dependency Learning Graph Neural Networks
• Time Series Analysis and Modeling to Forecast: a Survey
• A Neural Network Transformer Model for Composite Microstructure Homogenization
• Graph Deep Learning for Time Series Forecasting
• Neural Rough Differential Equations for Long Time Series
• NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting
• MAD: Self-Supervised Masked Anomaly Detection Task for Multivariate Time Series
• Bridging Physics-based and Data-driven modeling for Learning Dynamical Systems
• Adjusting for Autocorrelated Errors in Neural Networks for Time Series
• Domain Adaptation for Time Series Forecasting via Attention Sharing
• QuickSkill: Novice Skill Estimation in Online Multiplayer Games
• Blockwise Self-Attention for Long Document Understanding
• Monotonic Neural Ordinary Differential Equation: Time-series Forecasting for Cumulative Data
• SparseBERT: Rethinking the Importance Analysis in Self-attention
• TraDE: Transformers for Density Estimation
• Convolution-enhanced Evolving Attention Networks
• Protoformer: Embedding Prototypes for Transformers
• Unlocking the Potential of Deep Learning in Peak-Hour Series Forecasting
• End-to-End Modeling Hierarchical Time Series Using Autoregressive Transformer and Conditional Normalizing Flow based Reconciliation
• Differentiable Machine Learning-Based Modeling for Directly-Modulated Lasers
• Learning Differential Operators for Interpretable Time Series Modeling
• Hypergraph Learning based Recommender System for Anomaly Detection, Control and Optimization
• SigFormer: Signature Transformers for Deep Hedging
• Feature Selection for Multivariate Time Series via Network Pruning
• MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data
• Towards Better Long-range Time Series Forecasting using Generative Adversarial Networks
• Historical Inertia: A Neglected but Powerful Baseline for Long Sequence Time-series Forecasting
• Inter-Series Attention Model for COVID-19 Forecasting
• MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention
• Simulation-Informed Revenue Extrapolation with Confidence Estimate for Scaleup Companies Using Scarce Time-Series Data
• Multi-scale Transformer-based Network for Emotion Recognition from Multi Physiological Signals
• Real-time respiratory motion forecasting with online learning of recurrent neural networks for accurate targeting in externally guided radiotherapy
• The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
• ConvDySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention and Convolutional Neural Networks
• Deep Switching State Space Model (DS${}^3$M) for Nonlinear Time Series Forecasting with Regime Switching
• Correlation-aware Unsupervised Change-point Detection via Graph Neural Networks
• Dynamic Relational Inference in Multi-Agent Trajectories
• Meta-Forecasting by combining Global Deep Representations with Local Adaptation
• Neural forecasting at scale
• Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting
• Kernel Deformed Exponential Families for Sparse Continuous Attention