Cause and Effect: Hierarchical Concept-based Explanation of Neural Networks

Abstract

In many scenarios, human decisions are explained based on some high-level concepts. In this work, we take a step in the interpretability of neural networks by examining their internal representation or neuron's activations against concepts. A concept is characterized by a set of samples that have specific features in common. We propose a framework to check the existence of a causal relationship between a concept (or its negation) and task classes. While the previous methods focus on the importance of a concept to a task class, we go further and introduce four measures to quantitatively determine the order of causality. Moreover, we propose a method for constructing a hierarchy of concepts in the form of a concept-based decision tree which can shed light on how various concepts interact inside a neural network towards predicting output classes. Through experiments, we demonstrate the effectiveness of the proposed method in explaining the causal relationship between a concept and the predictive behaviour of a neural network as well as determining the interactions between different concepts through constructing a concept hierarchy.

arXiv comment13 pages, 14 figures

References in corpus (11)

• European Union regulations on algorithmic decision-making and a "right to explanation"
• The What-If Tool: Interactive Probing of Machine Learning Models
• Concept Whitening for Interpretable Image Recognition
• The (Un)reliability of saliency methods
• Concept Bottleneck Models
• Understanding intermediate layers using linear classifier probes
• Explaining Classifiers with Causal Concept Effect (CaCE)
• Entropy-based Logic Explanations of Neural Networks
• A Game-Theoretic Taxonomy of Visual Concepts in DNNs
• DISSECT: Disentangled Simultaneous Explanations via Concept Traversals
• Robust Semantic Interpretability: Revisiting Concept Activation Vectors