The Promise and Peril of Human Evaluation for Model Interpretability

Abstract

Transparency, user trust, and human comprehension are popular ethical motivations for interpretable machine learning. In support of these goals, researchers evaluate model explanation performance using humans and real world applications. This alone presents a challenge in many areas of artificial intelligence. In this position paper, we propose a distinction between descriptive and persuasive explanations. We discuss reasoning suggesting that functional interpretability may be correlated with cognitive function and user preferences. If this is indeed the case, evaluation and optimization using functional metrics could perpetuate implicit cognitive bias in explanations that threaten transparency. Finally, we propose two potential research directions to disambiguate cognitive function and explanation models, retaining control over the tradeoff between accuracy and interpretability.

Presented at NIPS 2017 Symposium on Interpretable Machine Learning. I'm not happy with the writing and presentation of these ideas and hope to submit an updated and extended version in 2020

References in corpus (4)

• Towards A Rigorous Science of Interpretable Machine Learning
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• Satisfying Real-world Goals with Dataset Constraints
• A Multi-Disciplinary Review of Knowledge Acquisition Methods: From Human to Autonomous Eliciting Agents

Cited by in corpus (14)

• A Survey on the Explainability of Supervised Machine Learning
• Machine Learning Testing: Survey, Landscapes and Horizons
• How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
• Quantifying Interpretability and Trust in Machine Learning Systems
• Evaluating Explanation Without Ground Truth in Interpretable Machine Learning
• Generating User-friendly Explanations for Loan Denials using GANs
• Micro-entries: Encouraging Deeper Evaluation of Mental Models Over Time for Interactive Data Systems
• Counterfactual Evaluation for Explainable AI
• A psychophysics approach for quantitative comparison of interpretable computer vision models
• Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context
• DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation
• Faithful and Plausible Explanations of Medical Code Predictions
• Quality Metrics for Transparent Machine Learning With and Without Humans In the Loop Are Not Correlated
• "A cold, technical decision-maker": Can AI provide explainability, negotiability, and humanity?