Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
arXiv:2010.12606
Abstract
Feature visualizations such as synthetic maximally activating images are a widely used explanation method to better understand the information processing of convolutional neural networks (CNNs). At the same time, there are concerns that these visualizations might not accurately represent CNNs' inner workings. Here, we measure how much extremely activating images help humans to predict CNN activations. Using a well-controlled psychophysical paradigm, we compare the informativeness of synthetic images by Olah et al. (2017) with a simple baseline visualization, namely exemplary natural images that also strongly activate a specific feature map. Given either synthetic or natural reference images, human participants choose which of two query images leads to strong positive activation. The experiments are designed to maximize participants' performance, and are the first to probe intermediate instead of final layer representations. We find that synthetic images indeed provide helpful information about feature map activations ( accuracy; chance would be ). However, natural images - originally intended as a baseline - outperform synthetic images by a wide margin (). Additionally, participants are faster and more confident for natural images, whereas subjective impressions about the interpretability of the feature visualizations are mixed. The higher informativeness of natural images holds across most layers, for both expert and lay participants as well as for hand- and randomly-picked feature visualizations. Even if only a single reference image is given, synthetic images provide less information than natural images ( vs. ). In summary, synthetic images from a popular feature visualization method are significantly less informative for assessing CNN activations than natural images. We argue that visualization methods should improve over this baseline.
arXiv commentPublished at ICLR 2021. Joint first and last authors. Code is available at https://bethgelab.github.io/testing_visualizations/
References in corpus (24)
- Distributed Representations of Words and Phrases and their Compositionality
- Towards A Rigorous Science of Interpretable Machine Learning
- Methods for Interpreting and Understanding Deep Neural Networks
- Interpretable machine learning: definitions, methods, and applications
- European Union regulations on algorithmic decision-making and a "right to explanation"
- Understanding Neural Networks Through Deep Visualization
- Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
- Visualizing and Understanding Recurrent Networks
- SmoothGrad: removing noise by adding noise
- Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
- A Benchmark for Interpretability Methods in Deep Neural Networks
- Understanding the Role of Individual Units in a Deep Neural Network
- Learning to Generate Reviews and Discovering Sentiment
- CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization
- On the importance of single directions for generalization
- Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
- Quantifying Interpretability and Trust in Machine Learning Systems
- It Takes Two to Tango: Towards Theory of AI's Mind
- Towards falsifiable interpretability research
- What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors
- Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
- Comparison-Based Framework for Psychophysics: Lab versus Crowdsourcing
- "I know it when I see it". Visualization and Intuitive Interpretability
- A psychophysics approach for quantitative comparison of interpretable computer vision models