Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Abstract

Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a $ProbSparse$ self-attention mechanism, which achieves $O(L\log L)$ in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

8 pages (main), 5 pages (appendix) and to be appeared in AAAI2021

References in corpus (22)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• Linformer: Self-Attention with Linear Complexity
• Effective Approaches to Attention-based Neural Machine Translation
• Generating Long Sequences with Sparse Transformers
• A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
• Reformer: The Efficient Transformer
• A Multi-Horizon Quantile Recurrent Forecaster
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• Long-term Forecasting using Higher Order Tensor RNNs
• A Memory-Network Based Solution for Multivariate Time-Series Forecasting
• Compressive Transformers for Long-Range Sequence Modelling
• CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation
• Adaptively Truncating Backpropagation Through Time to Control Gradient Bias
• ARMDN: Associative and Recurrent Mixture Density Networks for eRetail Demand Forecasting
• Blockwise Self-Attention for Long Document Understanding
• Approximate Bayesian Inference in Linear State Space Models for Intermittent Demand Forecasting at Scale
• DSTP-RNN: a dual-stage two-phase attention-based recurrent neural networks for long-term and multivariate time series prediction
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer

Cited by in corpus (11)

• TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting
• Dual-Branched Spatio-temporal Fusion Network for Multi-horizon Tropical Cyclone Track Forecast
• TVT: Transferable Vision Transformer for Unsupervised Domain Adaptation
• End-to-End User Behavior Retrieval in Click-Through RatePrediction Model
• Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention
• ADF & TransApp: A Transformer-Based Framework for Appliance Detection Using Smart Meter Consumption Series
• Few-Shot Optimization for Sensor Data Using Large Language Models: A Case Study on Fatigue Detection
• Multi-grained spatial-temporal feature complementarity for accurate online cellular traffic prediction
• Artificial Intelligence for Infectious Disease Prediction and Prevention: A Comprehensive Review
• Transfer training from smaller language model
• Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting