A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts
arXiv:2105.00290
Abstract
Despite substantial progress in applying neural networks (NN) to a wide variety of areas, they still largely suffer from a lack of transparency and interpretability. While recent developments in explainable artificial intelligence attempt to bridge this gap (e.g., by visualizing the correlation between input pixels and final outputs), these approaches are limited to explaining low-level relationships, and crucially, do not provide insights on error correction. In this work, we propose a framework (VRX) to interpret classification NNs with intuitive structural visual concepts. Given a trained classification model, the proposed VRX extracts relevant class-specific visual concepts and organizes them using structural concept graphs (SCG) based on pairwise concept relationships. By means of knowledge distillation, we show VRX can take a step towards mimicking the reasoning process of NNs and provide logical, concept-level explanations for final model decisions. With extensive experiments, we empirically show VRX can meaningfully answer "why" and "why not" questions about the prediction, providing easy-to-understand insights about the reasoning process. We also show that these insights can potentially provide guidance on improving NN's performance.
arXiv commentCVPR 2021
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Semi-Supervised Classification with Graph Convolutional Networks
- Inductive Representation Learning on Large Graphs
- Neural Message Passing for Quantum Chemistry
- Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
- Axiomatic Attribution for Deep Networks
- European Union regulations on algorithmic decision-making and a "right to explanation"
- Representation Learning on Graphs with Jumping Knowledge Networks
- Model-Agnostic Interpretability of Machine Learning
- Model compression via distillation and quantization
- Learning Conditioned Graph Structures for Interpretable Visual Question Answering
- Towards Understanding Knowledge Distillation