Explaining the Road Not Taken
arXiv:2103.14973
Abstract
It is unclear if existing interpretations of deep neural network models respond effectively to the needs of users. This paper summarizes the common forms of explanations (such as feature attribution, decision rules, or probes) used in over 200 recent papers about natural language processing (NLP), and compares them against user questions collected in the XAI Question Bank. We found that although users are interested in explanations for the road not taken -- namely, why the model chose one result and not a well-defined, seemly similar legitimate counterpart -- most model interpretations cannot answer these questions.
arXiv commentAccepted by The 2021 ACM CHI Workshop on Operationalizing Human-Centered Perspectives in Explainable AI (CHI 2021 HCXAI Workshop). For associated website, see https://human-centered-exnlp.github.io
References in corpus (10)
- Methods for Interpreting and Understanding Deep Neural Networks
- Understanding Black-box Predictions via Influence Functions
- e-SNLI: Natural Language Inference with Natural Language Explanations
- Pathologies of Neural Models Make Interpretations Difficult
- Self-Critical Reasoning for Robust Visual Question Answering
- Explaining Image Classifiers by Counterfactual Generation
- Representer Point Selection for Explaining Deep Neural Networks
- Multi-hop Question Answering via Reasoning Chains
- Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
- Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction