Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization

Abstract

Feature visualizations such as synthetic maximally activating images are a widely used explanation method to better understand the information processing of convolutional neural networks (CNNs). At the same time, there are concerns that these visualizations might not accurately represent CNNs' inner workings. Here, we measure how much extremely activating images help humans to predict CNN activations. Using a well-controlled psychophysical paradigm, we compare the informativeness of synthetic images by Olah et al. (2017) with a simple baseline visualization, namely exemplary natural images that also strongly activate a specific feature map. Given either synthetic or natural reference images, human participants choose which of two query images leads to strong positive activation. The experiments are designed to maximize participants' performance, and are the first to probe intermediate instead of final layer representations. We find that synthetic images indeed provide helpful information about feature map activations ($82\pm 4\%$ accuracy; chance would be $50\%$). However, natural images - originally intended as a baseline - outperform synthetic images by a wide margin ($92\pm 2\%$). Additionally, participants are faster and more confident for natural images, whereas subjective impressions about the interpretability of the feature visualizations are mixed. The higher informativeness of natural images holds across most layers, for both expert and lay participants as well as for hand- and randomly-picked feature visualizations. Even if only a single reference image is given, synthetic images provide less information than natural images ($65\pm 5\%$ vs. $73\pm 4\%$). In summary, synthetic images from a popular feature visualization method are significantly less informative for assessing CNN activations than natural images. We argue that visualization methods should improve over this baseline.

Published at ICLR 2021. Joint first and last authors. Code is available at https://bethgelab.github.io/testing_visualizations/

References in corpus (14)

• Towards A Rigorous Science of Interpretable Machine Learning
• Methods for Interpreting and Understanding Deep Neural Networks
• Understanding Neural Networks Through Deep Visualization
• SmoothGrad: removing noise by adding noise
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Understanding the Role of Individual Units in a Deep Neural Network
• CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization
• Quantifying Interpretability and Trust in Machine Learning Systems
• Towards falsifiable interpretability research
• What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
• Comparison-Based Framework for Psychophysics: Lab versus Crowdsourcing
• "I know it when I see it". Visualization and Intuitive Interpretability
• A psychophysics approach for quantitative comparison of interpretable computer vision models

Cited by in corpus (1)

• How Well do Feature Visualizations Support Causal Understanding of CNN Activations?