Model Explainability in Deep Learning Based Natural Language Processing
arXiv:2106.07410
Abstract
Machine learning (ML) model explainability has received growing attention, especially in the area related to model risk and regulations. In this paper, we reviewed and compared some popular ML model explainability methodologies, especially those related to Natural Language Processing (NLP) models. We then applied one of the NLP explainability methods Layer-wise Relevance Propagation (LRP) to a NLP classification model. We used the LRP method to derive a relevance score for each word in an instance, which is a local explainability. The relevance scores are then aggregated together to achieve global variable importance of the model. Through the case study, we also demonstrated how to apply the local explainability method to false positive and false negative instances to discover the weakness of a NLP model. These analysis can help us to understand NLP models better and reduce the risk due to the black-box nature of NLP models. We also identified some common issues due to the special natures of NLP models and discussed how explainability analysis can act as a control to detect these issues after the model has been trained.
arXiv comment12 pages, 8 figures
References in corpus (13)
- Methods for Interpreting and Understanding Deep Neural Networks
- Learning Important Features Through Propagating Activation Differences
- Model-Agnostic Interpretability of Machine Learning
- "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
- Explainable Recommendation via Multi-Task Learning in Opinionated Text Data
- Explainable Neural Networks based on Additive Index Models
- Locally Interpretable Models and Effects based on Supervised Partitioning (LIME-SUP)
- SHAP values for Explaining CNN-based Text Classification Models
- Towards Explainable NLP: A Generative Explanation Framework for Text Classification
- Understanding Convolutional Neural Networks for Text Classification
- Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State
- Model Interpretation: A Unified Derivative-based Framework for Nonparametric Regression and Supervised Machine Learning
- Analyzing and Interpreting Convolutional Neural Networks in NLP