GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
arXiv:1811.10597
Abstract
Generative Adversarial Networks (GANs) have recently achieved impressive results for many real-world applications, and many GAN variants have emerged with improvements in sample quality and training stability. However, they have not been well visualized or understood. How does a GAN represent our visual world internally? What causes the artifacts in GAN results? How do architectural choices affect GAN learning? Answering such questions could enable us to develop new insights and better models. In this work, we present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level. We first identify a group of interpretable units that are closely related to object concepts using a segmentation-based network dissection method. Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output. We examine the contextual relationship between these units and their surroundings by inserting the discovered object concepts into new images. We show several practical applications enabled by our framework, from comparing internal representations across different layers, models, and datasets, to improving GANs by locating and removing artifact-causing units, to interactively manipulating objects in a scene. We provide open source interpretation tools to help researchers and practitioners better understand their GAN models.
arXiv comment18 pages, 19 figures
References in corpus (8)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Spectral Normalization for Generative Adversarial Networks
- Axiomatic Attribution for Deep Networks
- Self-Attention Generative Adversarial Networks
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Visualizing and Understanding Recurrent Networks
- On the importance of single directions for generalization
Cited by in corpus (60)
- Understanding the Role of Individual Units in a Deep Neural Network
- A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
- Unsupervised Discovery of Interpretable Directions in the GAN Latent Space
- Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
- Controlling generative models with continuous factors of variations
- Closed-Form Factorization of Latent Semantics in GANs
- On Leveraging Pretrained GANs for Generation with Limited Data
- Understanding trained CNNs by indexing neuron selectivity
- Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
- Causality Learning: A New Perspective for Interpretable Machine Learning
- Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints
- On the "steerability" of generative adversarial networks
- Counterfactuals uncover the modular structure of deep generative models
- SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing
- InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
- Interpreting Super-Resolution Networks with Local Attribution Maps
- Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generation
- Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation
- Image Processing Using Multi-Code GAN Prior
- Vision and Language: from Visual Perception to Content Creation
- Editing in Style: Uncovering the Local Semantics of GANs
- RPGAN: GANs Interpretability via Random Routing
- Interpretation of Deep Temporal Representations by Selective Visualization of Internally Activated Nodes
- Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
- CoDeGAN: Contrastive Disentanglement for Generative Adversarial Network
- Conditional Image Generation and Manipulation for User-Specified Content
- MRI to PET Cross-Modality Translation using Globally and Locally Aware GAN (GLA-GAN) for Multi-Modal Diagnosis of Alzheimer's Disease
- Generative Hierarchical Features from Synthesizing Images
- Interventions and Counterfactuals in Tractable Probabilistic Models: Limitations of Contemporary Transformations
- Interpreting Face Inference Models using Hierarchical Network Dissection
- On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
- GAN Memory with No Forgetting
- GAN Inversion for Out-of-Range Images with Geometric Transformations
- Forging Emotions: a deep learning experiment on emotions and art
- Interpreting the Latent Space of GANs via Correlation Analysis for Controllable Concept Manipulation
- Interpolating GANs to Scaffold Autotelic Creativity
- Perceptually Validated Precise Local Editing for Facial Action Units with StyleGAN
- Interactive Mars Image Content-Based Search with Interpretable Machine Learning
- Ensembling with Deep Generative Views
- Diverse Image Generation via Self-Conditioned GANs
- Curriculum Learning for Deep Generative Models with Clustering
- Robust Training Using Natural Transformation
- Exploring Biases and Prejudice of Facial Synthesis via Semantic Latent Space
- Deep network as memory space: complexity, generalization, disentangled representation and interpretability
- Understanding of Kernels in CNN Models by Suppressing Irrelevant Visual Features in Images
- G3AN: Disentangling Appearance and Motion for Video Generation
- One-Shot Domain Adaptation For Face Generation
- TunaGAN: Interpretable GAN for Smart Editing
- Separating Content and Style for Unsupervised Image-to-Image Translation
- Face Images as Jigsaw Puzzles: Compositional Perception of Human Faces for Machines Using Generative Adversarial Networks
- Establishing an Evaluation Metric to Quantify Climate Change Image Realism
- Difference-in-Differences: Bridging Normalization and Disentanglement in PG-GAN
- MarioNette: Self-Supervised Sprite Learning
- Topology Maintained Structure Encoding
- TransferI2I: Transfer Learning for Image-to-Image Translation from Small Datasets
- Stay Positive: Non-Negative Image Synthesis for Augmented Reality
- Attack to Fool and Explain Deep Networks
- Controlled AutoEncoders to Generate Faces from Voices
- Inducing Hierarchical Compositional Model by Sparsifying Generator Network
- NeuroView: Explainable Deep Network Decision Making