Multi-Task Time Series Forecasting With Shared Attention

Abstract

Time series forecasting is a key component in many industrial and business decision processes and recurrent neural network (RNN) based models have achieved impressive progress on various time series forecasting tasks. However, most of the existing methods focus on single-task forecasting problems by learning separately based on limited supervised objectives, which often suffer from insufficient training instances. As the Transformer architecture and other attention-based models have demonstrated its great capability of capturing long term dependency, we propose two self-attention based sharing schemes for multi-task time series forecasting which can train jointly across multiple tasks. We augment a sequence of paralleled Transformer encoders with an external public multi-head attention function, which is updated by all data of all tasks. Experiments on a number of real-world multi-task time series forecasting tasks show that our proposed architectures can not only outperform the state-of-the-art single-task forecasting baselines but also outperform the RNN-based multi-task forecasting method.

arXiv commentAccepted by ICDMW 2020

References in corpus (7)

• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Decoupled Weight Decay Regularization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• WaveNet: A Generative Model for Raw Audio
• Meta Multi-Task Learning for Sequence Modeling