Evaluating Explanation Without Ground Truth in Interpretable Machine Learning

Abstract

Interpretable Machine Learning (IML) has become increasingly important in many real-world applications, such as autonomous cars and medical diagnosis, where explanations are significantly preferred to help people better understand how machine learning systems work and further enhance their trust towards systems. However, due to the diversified scenarios and subjective nature of explanations, we rarely have the ground truth for benchmark evaluation in IML on the quality of generated explanations. Having a sense of explanation quality not only matters for assessing system boundaries, but also helps to realize the true benefits to human users in practical settings. To benchmark the evaluation in IML, in this article, we rigorously define the problem of evaluating explanations, and systematically review the existing efforts from state-of-the-arts. Specifically, we summarize three general aspects of explanation (i.e., generalizability, fidelity and persuasibility) with formal definitions, and respectively review the representative methodologies for each of them under different tasks. Further, a unified evaluation framework is designed according to the hierarchical needs from developers and end-users, which could be easily adopted for different scenarios in practice. In the end, open problems are discussed, and several limitations of current evaluation techniques are raised for future explorations.

References in corpus (22)

• Distilling the Knowledge in a Neural Network
• Towards A Rigorous Science of Interpretable Machine Learning
• Methods for Interpreting and Understanding Deep Neural Networks
• Interpretable machine learning: definitions, methods, and applications
• Understanding Black-box Predictions via Influence Functions
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Actionable Recourse in Linear Classification
• Distilling a Neural Network Into a Soft Decision Tree
• An Evaluation of the Human-Interpretability of Explanation
• Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
• Tell Me Where to Look: Guided Attention Inference Network
• Interpretation of Neural Networks is Fragile
• On the (In)fidelity and Sensitivity for Explanations
• The Promise and Peril of Human Evaluation for Model Interpretability
• Explaining Deep Learning Models using Causal Inference
• Interpreting CNNs via Decision Trees
• What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
• Interpretable Active Learning
• Learning Interpretable Models with Causal Guarantees
• Deriving Machine Attention from Human Rationales
• Interpreting Neural Networks Using Flip Points

Cited by in corpus (12)

• Explainable Matrix -- Visualization for Global and Local Interpretability of Random Forest Classification Ensembles
• On Completeness-aware Concept-Based Explanations in Deep Neural Networks
• Quantitative Evaluations on Saliency Methods: An Experimental Study
• Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation
• Explainable Rumor Detection using Inter and Intra-feature Attention Networks
• Towards a Unified Evaluation of Explanation Methods without Ground Truth
• Data Representing Ground-Truth Explanations to Evaluate XAI Methods
• Learning Credible Deep Neural Networks with Rationale Regularization
• Explainable Deep Relational Networks for Predicting Compound-Protein Affinities and Contacts
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• Consensus-based Interpretable Deep Neural Networks with Application to Mortality Prediction
• Developing a Fidelity Evaluation Approach for Interpretable Machine Learning