Explaining and Interpreting LSTMs

Abstract

While neural networks have acted as a strong unifying force in the design of modern AI systems, the neural network architectures themselves remain highly heterogeneous due to the variety of tasks to be solved. In this chapter, we explore how to adapt the Layer-wise Relevance Propagation (LRP) technique used for explaining the predictions of feed-forward networks to the LSTM architecture used for sequential data modeling and forecasting. The special accumulators and gated interactions present in the LSTM require both a new propagation scheme and an extension of the underlying theoretical framework to deliver faithful explanations.

28 pages, 7 figures, book chapter, In: Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, LNCS volume 11700, Springer 2019. arXiv admin note: text overlap with arXiv:1806.07857

References in corpus (8)

• Deep Learning in Neural Networks: An Overview
• LSTM: A Search Space Odyssey
• Methods for Interpreting and Understanding Deep Neural Networks
• Asynchronous Methods for Deep Reinforcement Learning
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
• "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
• Explaining the Unique Nature of Individual Gait Patterns with Deep Learning

Cited by in corpus (14)

• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Towards Explainable Artificial Intelligence
• Physically Interpretable Neural Networks for the Geosciences: Applications to Earth System Variability
• Higher-Order Explanations of Graph Neural Networks via Relevant Walks
• On the Explainability of Natural Language Processing Deep Models
• From Clustering to Cluster Explanations via Neural Networks
• Toward Explainable AI for Regression Models
• Opening the Black-Box: A Systematic Review on Explainable AI in Remote Sensing
• The Clever Hans Effect in Unsupervised Learning
• XNAP: Making LSTM-based Next Activity Predictions Explainable by Using LRP
• Analyzing Atomic Interactions in Molecules as Learned by Neural Networks
• Interpretability is in the eye of the beholder: Human versus artificial classification of image segments generated by humans versus XAI
• The benefits and costs of explainable artificial intelligence in visual quality control: Evidence from fault detection performance and eye movements
• Satellite galaxies' drag on field stars in the Milky Way