Exploring Interpretable LSTM Neural Networks over Multi-Variable Data

Abstract

For recurrent neural networks trained on time series with target and exogenous variables, in addition to accurate prediction, it is also desired to provide interpretable insights into the data. In this paper, we explore the structure of LSTM recurrent neural networks to learn variable-wise hidden states, with the aim to capture different dynamics in multi-variable time series and distinguish the contribution of variables to the prediction. With these variable-wise hidden states, a mixture attention mechanism is proposed to model the generative process of the target. Then we develop associated training methods to jointly learn network parameters, variable and temporal importance w.r.t the prediction of the target variable. Extensive experiments on real datasets demonstrate enhanced prediction performance by capturing the dynamics of different variables. Meanwhile, we evaluate the interpretation results both qualitatively and quantitatively. It exhibits the prospect as an end-to-end framework for both forecasting and knowledge extraction over multi-variable data.

Accepted to International Conference on Machine Learning (ICML), 2019

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Methods for Interpreting and Understanding Deep Neural Networks
• Learning Important Features Through Propagating Activation Differences
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Learning to Diagnose with LSTM Recurrent Neural Networks
• A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Factorization tricks for LSTM networks
• Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks
• Focused Hierarchical RNNs for Conditional Sequence Processing
• Learning Deep Matrix Representations

Cited by in corpus (15)

• Neural Granger Causality
• Interpreting Deep Learning-Based Networking Systems
• Adaptive Dependency Learning Graph Neural Networks
• Predicting machine failures from multivariate time series: an industrial case study
• Explainable Tensorized Neural Ordinary Differential Equations forArbitrary-step Time Series Prediction
• Modality Fusion Network and Personalized Attention in Momentary Stress Detection in the Wild
• Sequential Interpretability: Methods, Applications, and Future Direction for Understanding Deep Learning Models in the Context of Sequential Data
• What went wrong and when? Instance-wise Feature Importance for Time-series Models
• Mixture of Linear Models Co-supervised by Deep Neural Networks
• Attention Mechanism for Multivariate Time Series Recurrent Model Interpretability Applied to the Ironmaking Industry
• Making Good on LSTMs' Unfulfilled Promise
• Interpretable Additive Recurrent Neural Networks For Multivariate Clinical Time Series
• Temporal mixture ensemble models for intraday volume forecasting in cryptocurrency exchange markets
• A Knowledge Distillation Ensemble Framework for Predicting Short and Long-term Hospitalisation Outcomes from Electronic Health Records Data
• Discovering long term dependencies in noisy time series data using deep learning