How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation

Abstract

Recent years have seen a boom in interest in machine learning systems that can provide a human-understandable rationale for their predictions or decisions. However, exactly what kinds of explanation are truly human-interpretable remains poorly understood. This work advances our understanding of what makes explanations interpretable in the specific context of verification. Suppose we have a machine learning system that predicts X, and we provide rationale for this prediction X. Given an input, an explanation, and an output, is the output consistent with the input and the supposed rationale? Via a series of user-studies, we identify what kinds of increases in complexity have the greatest effect on the time it takes for humans to verify the rationale, and which seem relatively insensitive.

References in corpus (9)

• Towards A Rigorous Science of Interpretable Machine Learning
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• SmoothGrad: removing noise by adding noise
• Supersparse Linear Integer Models for Optimized Medical Scoring Systems
• Learning how to explain neural networks: PatternNet and PatternAttribution
• Increasing the Interpretability of Recurrent Neural Networks Using Hidden Markov Models
• The Promise and Peril of Human Evaluation for Model Interpretability
• Programs as Black-Box Explanations
• Supervised topic models for clinical interpretability

Cited by in corpus (26)

• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
• Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
• Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
• A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models
• How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations
• Benchmarking Attribution Methods with Relative Feature Importance
• The Impact of Explanations on Layperson Trust in Artificial Intelligence-Driven Symptom Checker Apps: Experimental Study
• Deep Weighted Averaging Classifiers
• Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
• Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
• Personalized explanation in machine learning: A conceptualization
• Generating User-friendly Explanations for Loan Denials using GANs
• VINE: Visualizing Statistical Interactions in Black Box Models
• Artificial Intelligence Strategies for National Security and Safety Standards
• Do Explanations make VQA Models more Predictable to a Human?
• A Study on Multimodal and Interactive Explanations for Visual Question Answering
• Adversarial Attacks and Defenses: An Interpretation Perspective
• Auditing and Debugging Deep Learning Models via Decision Boundaries: Individual-level and Group-level Analysis
• On Two XAI Cultures: A Case Study of Non-technical Explanations in Deployed AI System
• Crowdsourcing Evaluation of Saliency-based XAI Methods
• Towards Explainable Fact Checking
• Understanding Information Processing in Human Brain by Interpreting Machine Learning Models