Understanding the Role of Individual Units in a Deep Neural Network
arXiv:2009.05041 · doi:10.1073/pnas.1907375117
Abstract
Deep neural networks excel at finding hierarchical representations that solve complex tasks over large data sets. How can we humans understand these learned representations? In this work, we present network dissection, an analytic framework to systematically identify the semantics of individual hidden units within image classification and image generation networks. First, we analyze a convolutional neural network (CNN) trained on scene classification and discover units that match a diverse set of object concepts. We find evidence that the network has learned many object classes that play crucial roles in classifying scene classes. Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes. By analyzing changes made when small sets of units are activated or deactivated, we find that objects can be added and removed from the output scenes while adapting to the context. Finally, we apply our analytic framework to understanding adversarial attacks and to semantic image editing.
arXiv commentProceedings of the National Academy of Sciences 2020. Code at https://github.com/davidbau/dissect/ and website at https://dissect.csail.mit.edu/
References in corpus (18)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- A Unified Approach to Interpreting Model Predictions
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Intriguing properties of neural networks
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
- Visualizing and Understanding Recurrent Networks
- SmoothGrad: removing noise by adding noise
- ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
- Understanding the Role of Individual Units in a Deep Neural Network
- Learning to Generate Reviews and Discovering Sentiment
- On the importance of single directions for generalization
- RISE: Randomized Input Sampling for Explanation of Black-box Models
- GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
- Revisiting the Importance of Individual Units in CNNs via Ablation
- Interpreting Deep Visual Representations via Network Dissection
- Learning Finite State Representations of Recurrent Policy Networks
Cited by in corpus (47)
- Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
- Understanding the Role of Individual Units in a Deep Neural Network
- From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
- Acquisition of Chess Knowledge in AlphaZero
- Barbershop: GAN-based Image Compositing using Segmentation Masks
- Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment
- On Interpretability of Artificial Neural Networks: A Survey
- Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
- Meta-trained agents implement Bayes-optimal agents
- Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
- Human-like object concept representations emerge naturally in multimodal large language models
- Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
- Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification
- Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems
- InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
- Unsupervised Point Cloud Pre-Training via Occlusion Completion
- The Role of Edges in Line Drawing Perception
- Towards Counterfactual and Contrastive Explainability and Transparency of DCNN Image Classifiers
- Deeply Explain CNN via Hierarchical Decomposition
- Emergence of Concepts in DNNs?
- Distributional Generalization: A New Kind of Generalization
- Interpreting intermediate convolutional layers of generative CNNs trained on waveforms
- HyperInverter: Improving StyleGAN Inversion via Hypernetwork
- Interpretable Deep Feature Propagation for Early Action Recognition
- Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories
- PCE-PINNs: Physics-Informed Neural Networks for Uncertainty Propagation in Ocean Modeling
- Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
- Preserve, Promote, or Attack? GNN Explanation via Topology Perturbation
- On the Value of Labeled Data and Symbolic Methods for Hidden Neuron Activation Analysis
- Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability
- AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
- Leveraging Sparse Linear Layers for Debuggable Deep Networks
- Interpreting Face Inference Models using Hierarchical Network Dissection
- Auditing AI models for Verified Deployment under Semantic Specifications
- DeepEverest: Accelerating Declarative Top-K Queries for Deep Neural Network Interpretation
- Fast gradient-free activation maximization for neurons in spiking neural networks
- Visualization for Multivariate Gaussian Anomaly Detection in Images
- Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation
- Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Space
- Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
- HOLMES: HOLonym-MEronym based Semantic inspection for Convolutional Image Classifiers
- IFBiD: Inference-Free Bias Detection
- Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
- Neuron Coverage-Guided Domain Generalization
- Task Guided Compositional Representation Learning for ZDA
- NeuroView: Explainable Deep Network Decision Making
- Caption-Driven Explainability: Probing CNNs for Bias via CLIP