A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction

Abstract

The Nonlinear autoregressive exogenous (NARX) model, which predicts the current value of a time series based upon its previous values as well as the current and past values of multiple driving (exogenous) series, has been studied for decades. Despite the fact that various NARX models have been developed, few of them can capture the long-term temporal dependencies appropriately and select the relevant driving series to make predictions. In this paper, we propose a dual-stage attention-based recurrent neural network (DA-RNN) to address these two issues. In the first stage, we introduce an input attention mechanism to adaptively extract relevant driving series (a.k.a., input features) at each time step by referring to the previous encoder hidden state. In the second stage, we use a temporal attention mechanism to select relevant encoder hidden states across all time steps. With this dual-stage attention scheme, our model can not only make predictions effectively, but can also be easily interpreted. Thorough empirical studies based upon the SML 2010 dataset and the NASDAQ 100 Stock dataset demonstrate that the DA-RNN can outperform state-of-the-art methods for time series prediction.

International Joint Conference on Artificial Intelligence (IJCAI), 2017

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Cited by in corpus (77)

• Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions
• Applications of deep learning in stock market prediction: recent progress
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Parallel Spatio-Temporal Attention-Based TCN for Multivariate Time Series Prediction
• Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis
• Crop Yield Prediction Integrating Genotype and Weather Variables Using Deep Learning
• N-BEATS: Neural basis expansion analysis for interpretable time series forecasting
• A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data
• Exploring Interpretable LSTM Neural Networks over Multi-Variable Data
• Long-Range Transformers for Dynamic Spatiotemporal Forecasting
• BuildSenSys: Reusing Building Sensing Data for Traffic Prediction with Cross-domain Learning
• MGP-AttTCN: An Interpretable Machine Learning Model for the Prediction of Sepsis
• A Joint Model for IT Operation Series Prediction and Anomaly Detection
• A Memory-Network Based Solution for Multivariate Time-Series Forecasting
• Non-linear Functional Modeling using Neural Networks
• Roadmap on machine learning glassy dynamics
• AdaRNN: Adaptive Learning and Forecasting of Time Series
• AlphaEvolve: A Learning Framework to Discover Novel Alphas in Quantitative Investment
• EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction
• Hierarchical Attention-Based Recurrent Highway Networks for Time Series Prediction
• DP-LSTM: Differential Privacy-inspired LSTM for Stock Prediction Using Financial News
• Medical Time Series Classification with Hierarchical Attention-based Temporal Convolutional Networks: A Case Study of Myotonic Dystrophy Diagnosis
• Adjusting for Autocorrelated Errors in Neural Networks for Time Series
• HyperST-Net: Hypernetworks for Spatio-Temporal Forecasting
• Multi-variable LSTM neural network for autoregressive exogenous model
• Do We Really Need Deep Learning Models for Time Series Forecasting?
• ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models
• Joint Modeling of Local and Global Temporal Dynamics for Multivariate Time Series Forecasting with Missing Values
• Demystifying Deep Learning in Predictive Spatio-Temporal Analytics: An Information-Theoretic Framework
• Risk Management via Anomaly Circumvent: Mnemonic Deep Learning for Midterm Stock Prediction
• Forecasting Individualized Disease Trajectories using Interpretable Deep Learning
• A Multi-view Multi-task Learning Framework for Multi-variate Time Series Forecasting
• Temporal-Relational Hypergraph Tri-Attention Networks for Stock Trend Prediction
• From Known to Unknown: Knowledge-guided Transformer for Time-Series Sales Forecasting in Alibaba
• Knowledge Adaption for Demand Prediction based on Multi-task Memory Neural Network
• Alpha Discovery Neural Network based on Prior Knowledge
• Multilevel Wavelet Decomposition Network for Interpretable Time Series Analysis
• Learning representations for multivariate time series with missing data using Temporal Kernelized Autoencoders
• Attention Mechanism for Multivariate Time Series Recurrent Model Interpretability Applied to the Ironmaking Industry
• Parallel Extraction of Long-term Trends and Short-term Fluctuation Framework for Multivariate Time Series Forecasting
• Feature Selection for Multivariate Time Series via Network Pruning
• Non-Gaussian Gaussian Processes for Few-Shot Regression
• Transformers for Limit Order Books
• Towards Better Forecasting by Fusing Near and Distant Future Visions
• Particle Filter Recurrent Neural Networks
• Transform-Based Multilinear Dynamical System for Tensor Time Series Analysis
• Multi-Task Prediction of Clinical Outcomes in the Intensive Care Unit using Flexible Multimodal Transformers
• Fast ES-RNN: A GPU Implementation of the ES-RNN Algorithm
• daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery
• TENT: Tensorized Encoder Transformer for Temperature Forecasting
• Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting
• Improved Predictive Deep Temporal Neural Networks with Trend Filtering
• Leveraging Multiple Relations for Fashion Trend Forecasting Based on Social Media
• Spatiotemporal Adaptive Neural Network for Long-term Forecasting of Financial Time Series
• Structural Forecasting for Tropical Cyclone Intensity Prediction: Providing Insight with Deep Learning
• CASTNet: Community-Attentive Spatio-Temporal Networks for Opioid Overdose Forecasting
• TRACER: A Framework for Facilitating Accurate and Interpretable Analytics for High Stakes Applications
• Learning Multiple Stock Trading Patterns with Temporal Routing Adaptor and Optimal Transport
• FlowDB a large scale precipitation, river, and flash flood dataset
• Wavelet Denoising and Attention-based RNN-ARIMA Model to Predict Forex Price
• Real time expert system for anomaly detection of aerators based on computer vision technology and existing surveillance cameras
• Optimization of Wireless Sensor Network Deployment for Spatiotemporal Reconstruction and Prediction
• Temporal Pattern Attention for Multivariate Time Series Forecasting
• Explainable Text-Driven Neural Network for Stock Prediction
• Latent Variable Algorithms for Multimodal Learning and Sensor Fusion
• Hybrid Attention Networks for Flow and Pressure Forecasting in Water Distribution Systems
• DDLSTM: Dual-Domain LSTM for Cross-Dataset Action Recognition
• Relation-aware Meta-learning for Market Segment Demand Prediction with Limited Records
• Visualising Deep Network's Time-Series Representations
• HECT: High-Dimensional Ensemble Consistency Testing for Climate Models
• AI-enabled Prediction of eSports Player Performance Using the Data from Heterogeneous Sensors
• Graph Deep Factors for Forecasting
• Learning Invariant Representation of Tasks for Robust Surgical State Estimation
• Introducing a Generative Adversarial Network Model for Lagrangian Trajectory Simulation
• Dynamic Gaussian Mixture based Deep Generative Model For Robust Forecasting on Sparse Multivariate Time Series
• Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting