DeepSeer: Interactive RNN Explanation and Debugging via State Abstraction
arXiv:2303.01576 · doi:10.1145/3544548.3580852
Abstract
Recurrent Neural Networks (RNNs) have been widely used in Natural Language Processing (NLP) tasks given its superior performance on processing sequential data. However, it is challenging to interpret and debug RNNs due to the inherent complexity and the lack of transparency of RNNs. While many explainable AI (XAI) techniques have been proposed for RNNs, most of them only support local explanations rather than global explanations. In this paper, we present DeepSeer, an interactive system that provides both global and local explanations of RNN behavior in multiple tightly-coordinated views for model understanding and debugging. The core of DeepSeer is a state abstraction method that bundles semantically similar hidden states in an RNN model and abstracts the model as a finite state machine. Users can explore the global model behavior by inspecting text patterns associated with each state and the transitions between states. Users can also dive into individual predictions by inspecting the state trace and intermediate prediction results of a given input. A between-subjects user study with 28 participants shows that, compared with a popular XAI technique, LIME, participants using DeepSeer made a deeper and more comprehensive assessment of RNN model behavior, identified the root causes of incorrect predictions more accurately, and came up with more actionable plans to improve the model performance.
arXiv commentTo appear in the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23), April 23--28, 2023, Hamburg, Germany
References in corpus (13)
- A Unified Approach to Interpreting Model Predictions
- Character-level Convolutional Networks for Text Classification
- Methods for Interpreting and Understanding Deep Neural Networks
- Understanding Black-box Predictions via Influence Functions
- Visualizing and Understanding Recurrent Networks
- Questioning the AI: Informing Design Practices for Explainable AI User Experiences
- Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
- Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
- Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
- Extracting Automata from Recurrent Neural Networks Using Queries and Counterexamples
- On the Accuracy of Influence Functions for Measuring Group Effects
- A study of data and label shift in the LIME framework
- GNNLens: A Visual Analytics Approach for Prediction Error Diagnosis of Graph Neural Networks