Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction

Abstract

We present a randomized controlled trial for a model-in-the-loop regression task, with the goal of measuring the extent to which (1) good explanations of model predictions increase human accuracy, and (2) faulty explanations decrease human trust in the model. We study explanations based on visual saliency in an image-based age prediction task for which humans and learned models are individually capable but not highly proficient and frequently disagree. Our experimental design separates model quality from explanation quality, and makes it possible to compare treatments involving a variety of explanations of varying levels of quality. We find that presenting model predictions improves human accuracy. However, visual explanations of various kinds fail to significantly alter human accuracy or trust in the model - regardless of whether explanations characterize an accurate model, an inaccurate one, or are generated randomly and independently of the input image. These findings suggest the need for greater evaluation of explanations in downstream decision making tasks, better design-based tools for presenting explanations to users, and better approaches for generating explanations.

References in corpus (10)

• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Wide Residual Networks
• Deep Learning for Identifying Metastatic Breast Cancer
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
• How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
• It Takes Two to Tango: Towards Theory of AI's Mind
• Investigating Human + Machine Complementarity for Recidivism Predictions

Cited by in corpus (10)

• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
• Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
• Debugging Tests for Model Explanations
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels
• Intelligent Decision Assistance Versus Automated Decision-Making: Enhancing Knowledge Work Through Explainable Artificial Intelligence
• Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments
• Explaining the Road Not Taken
• Do Input Gradients Highlight Discriminative Features?