Understanding the Role of Individual Units in a Deep Neural Network

Abstract

Deep neural networks excel at finding hierarchical representations that solve complex tasks over large data sets. How can we humans understand these learned representations? In this work, we present network dissection, an analytic framework to systematically identify the semantics of individual hidden units within image classification and image generation networks. First, we analyze a convolutional neural network (CNN) trained on scene classification and discover units that match a diverse set of object concepts. We find evidence that the network has learned many object classes that play crucial roles in classifying scene classes. Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes. By analyzing changes made when small sets of units are activated or deactivated, we find that objects can be added and removed from the output scenes while adapting to the context. Finally, we apply our analytic framework to understanding adversarial attacks and to semantic image editing.

Proceedings of the National Academy of Sciences 2020. Code at https://github.com/davidbau/dissect/ and website at https://dissect.csail.mit.edu/

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SmoothGrad: removing noise by adding noise
• Understanding the Role of Individual Units in a Deep Neural Network

Cited by in corpus (39)

• Understanding the Role of Individual Units in a Deep Neural Network
• From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
• Acquisition of Chess Knowledge in AlphaZero
• Barbershop: GAN-based Image Compositing using Segmentation Masks
• Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment
• Meta-trained agents implement Bayes-optimal agents
• Human-like object concept representations emerge naturally in multimodal large language models
• Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
• Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification
• Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems
• InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
• Towards Counterfactual and Contrastive Explainability and Transparency of DCNN Image Classifiers
• Deeply Explain CNN via Hierarchical Decomposition
• The Role of Edges in Line Drawing Perception
• Emergence of Concepts in DNNs?
• Interpreting intermediate convolutional layers of generative CNNs trained on waveforms
• HyperInverter: Improving StyleGAN Inversion via Hypernetwork
• Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories
• Interpretable Deep Feature Propagation for Early Action Recognition
• PCE-PINNs: Physics-Informed Neural Networks for Uncertainty Propagation in Ocean Modeling
• Preserve, Promote, or Attack? GNN Explanation via Topology Perturbation
• On the Value of Labeled Data and Symbolic Methods for Hidden Neuron Activation Analysis
• Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability
• Auditing AI models for Verified Deployment under Semantic Specifications
• AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
• Interpreting Face Inference Models using Hierarchical Network Dissection
• Leveraging Sparse Linear Layers for Debuggable Deep Networks
• Fast gradient-free activation maximization for neurons in spiking neural networks
• DeepEverest: Accelerating Declarative Top-K Queries for Deep Neural Network Interpretation
• Visualization for Multivariate Gaussian Anomaly Detection in Images
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
• Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space
• Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation
• Caption-Driven Explainability: Probing CNNs for Bias via CLIP
• HOLMES: HOLonym-MEronym based Semantic inspection for Convolutional Image Classifiers
• Task Guided Compositional Representation Learning for ZDA
• Neuron Coverage-Guided Domain Generalization
• IFBiD: Inference-Free Bias Detection
• NeuroView: Explainable Deep Network Decision Making