Evaluating the visualization of what a Deep Neural Network has learned

Abstract

Deep Neural Networks (DNNs) have demonstrated impressive performance in complex machine learning tasks such as image classification or speech recognition. However, due to their multi-layer nonlinear structure, they are not transparent, i.e., it is hard to grasp what makes them arrive at a particular classification or recognition decision given a new unseen data sample. Recently, several approaches have been proposed enabling one to understand and interpret the reasoning embodied in a DNN for a single test image. These methods quantify the ''importance'' of individual pixels wrt the classification decision and allow a visualization in terms of a heatmap in pixel/input space. While the usefulness of heatmaps can be judged subjectively by a human, an objective quality measure is missing. In this paper we present a general methodology based on region perturbation for evaluating ordered collections of pixels such as heatmaps. We compare heatmaps computed by three different methods on the SUN397, ILSVRC2012 and MIT Places data sets. Our main result is that the recently proposed Layer-wise Relevance Propagation (LRP) algorithm qualitatively and quantitatively provides a better explanation of what made a DNN arrive at a particular classification decision than the sensitivity-based approach or the deconvolution method. We provide theoretical arguments to explain this result and discuss its practical implications. Finally, we investigate the use of heatmaps for unsupervised assessment of neural network performance.

13 pages, 8 Figures

References in corpus (5)

• Explaining and Harnessing Adversarial Examples
• Natural Language Processing (almost) from Scratch
• Understanding Neural Networks Through Deep Visualization
• Object Detectors Emerge in Deep Scene CNNs
• Understanding Deep Image Representations by Inverting Them

Cited by in corpus (10)

• Axiomatic Attribution for Deep Networks
• Layer-Wise Relevance Propagation for Explaining Deep Neural Network Decisions in MRI-Based Alzheimer's Disease Classification
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• How Important Is a Neuron?
• VisualBackProp: efficient visualization of CNNs
• Feature Importance Measure for Non-linear Learning Algorithms
• Feature Visualization within an Automated Design Assessment leveraging Explainable Artificial Intelligence Methods
• Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
• LAMVI-2: A Visual Tool for Comparing and Tuning Word Embedding Models
• Understanding Convolutional Neural Networks with A Mathematical Model