MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention
arXiv:2009.14799
Abstract
Recent advances in neural forecasting have produced major improvements in accuracy for probabilistic demand prediction. In this work, we propose novel improvements to the current state of the art by incorporating changes inspired by recent advances in Transformer architectures for Natural Language Processing. We develop a novel decoder-encoder attention for context-alignment, improving forecasting accuracy by allowing the network to study its own history based on the context for which it is producing a forecast. We also present a novel positional encoding that allows the neural network to learn context-dependent seasonality functions as well as arbitrary holiday distances. Finally we show that the current state of the art MQ-Forecaster (Wen et al., 2017) models display excess variability by failing to leverage previous errors in the forecast to improve accuracy. We propose a novel decoder-self attention scheme for forecasting that produces significant improvements in the excess variation of the forecast.
References in corpus (11)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
- Attention in Natural Language Processing
- A Multi-Horizon Quantile Recurrent Forecaster
- Long Short-Term Memory-Networks for Machine Reading
- STConvS2S: Spatiotemporal Convolutional Sequence to Sequence Network for Weather Forecasting
- Long-term Forecasting using Higher Order Tensor RNNs
- Financial series prediction using Attention LSTM
- Deep Generative Quantile-Copula Models for Probabilistic Forecasting