How Well do Feature Visualizations Support Causal Understanding of CNN Activations?

Abstract

A precise understanding of why units in an artificial network respond to certain stimuli would constitute a big step towards explainable artificial intelligence. One widely used approach towards this goal is to visualize unit responses via activation maximization. These synthetic feature visualizations are purported to provide humans with precise information about the image features that cause a unit to be activated - an advantage over other alternatives like strongly activating natural dataset samples. If humans indeed gain causal insight from visualizations, this should enable them to predict the effect of an intervention, such as how occluding a certain patch of the image (say, a dog's head) changes a unit's activation. Here, we test this hypothesis by asking humans to decide which of two square occlusions causes a larger change to a unit's activation. Both a large-scale crowdsourced experiment and measurements with experts show that on average the extremely activating feature visualizations by Olah et al. (2017) indeed help humans on this task ($68\pm 4$% accuracy; baseline performance without any visualizations is $60\pm 3$%). However, they do not provide any substantial advantage over other visualizations (such as e.g. dataset samples), which yield similar performance ($66\pm 3$% to $67\pm 3$% accuracy). Taken together, we propose an objective psychophysical task to quantify the benefit of unit-level interpretability methods for humans, and find no evidence that a widely-used feature visualization method provides humans with better "causal understanding" of unit activations than simple alternative visualizations.

Presented at NeurIPS 2021. Shared first and last authorship. Project website at https://brendel-group.github.io/causal-understanding-via-visualizations/

References in corpus (11)

• Towards A Rigorous Science of Interpretable Machine Learning
• Preserving Causal Constraints in Counterfactual Explanations for Machine Learning Classifiers
• To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods
• Are we done with ImageNet?
• From ImageNet to Image Classification: Contextualizing Progress on Benchmarks
• A neural network walks into a lab: towards using deep nets as models for human behavior
• Towards falsifiable interpretability research
• Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
• Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• A psychophysics approach for quantitative comparison of interpretable computer vision models