An Evaluation of the Human-Interpretability of Explanation

Abstract

Recent years have seen a boom in interest in machine learning systems that can provide a human-understandable rationale for their predictions or decisions. However, exactly what kinds of explanation are truly human-interpretable remains poorly understood. This work advances our understanding of what makes explanations interpretable under three specific tasks that users may perform with machine learning systems: simulation of the response, verification of a suggested response, and determining whether the correctness of a suggested response changes under a change to the inputs. Through carefully controlled human-subject experiments, we identify regularizers that can be used to optimize for the interpretability of machine learning systems. Our results show that the type of complexity matters: cognitive chunks (newly defined concepts) affect performance more than variable repetitions, and these trends are consistent across tasks and domains. This suggests that there may exist some common design principles for explanation systems.

arXiv commentarXiv admin note: substantial text overlap with arXiv:1802.00682

References in corpus (8)

• Towards A Rigorous Science of Interpretable Machine Learning
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• SmoothGrad: removing noise by adding noise
• Supersparse Linear Integer Models for Optimized Medical Scoring Systems
• Learning how to explain neural networks: PatternNet and PatternAttribution
• Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models
• Programs as Black-Box Explanations
• Supervised topic models for clinical interpretability

Cited by in corpus (37)

• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• A Survey on the Explainability of Supervised Machine Learning
• The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
• Towards Explainable Artificial Intelligence
• A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
• A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models
• Algorithmic Recourse: from Counterfactual Explanations to Interventions
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• Assessing the Local Interpretability of Machine Learning Models
• Explain and Predict, and then Predict Again
• A survey of algorithmic recourse: definitions, formulations, solutions, and prospects
• Evaluating Explanation Without Ground Truth in Interpretable Machine Learning
• "How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
• A Human-Grounded Evaluation of SHAP for Alert Processing
• DeepView: Visualizing Classification Boundaries of Deep Neural Networks as Scatter Plots Using Discriminative Dimensionality Reduction
• Evaluation of Similarity-based Explanations
• Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond
• Trusting RoBERTa over BERT: Insights from CheckListing the Natural Language Inference Task
• Towards a Unified Multidimensional Explainability Metric: Evaluating Trustworthiness in AI Models
• Information Fusion via Symbolic Regression: A Tutorial in the Context of Human Health
• Does Your Model Think Like an Engineer? Explainable AI for Bearing Fault Detection with Deep Learning
• Harnessing Explanations to Bridge AI and Humans
• Optimal Transport for Efficient, Unsupervised Anomaly Detection on Industrial Data
• Challenging common interpretability assumptions in feature attribution explanations
• A Review of Explainable Artificial Intelligence in Manufacturing
• Domain-Level Explainability -- A Challenge for Creating Trust in Superhuman AI Strategies
• Auditing and Debugging Deep Learning Models via Decision Boundaries: Individual-level and Group-level Analysis
• What Would You Ask the Machine Learning Model? Identification of User Needs for Model Explanations Based on Human-Model Conversations
• Biased Models Have Biased Explanations
• Explaining Reward Functions to Humans for Better Human-Robot Collaboration
• One-shot Policy Elicitation via Semantic Reward Manipulation
• Explainable AI for medical imaging: Explaining pneumothorax diagnoses with Bayesian Teaching
• Visualizing Rule Sets: Exploration and Validation of a Design Space
• Explaining Creative Artifacts
• Learnt Sparsification for Interpretable Graph Neural Networks
• Scalable Explanation of Inferences on Large Graphs
• An Exploration And Validation of Visual Factors in Understanding Classification Rule Sets