Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

Abstract

Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what generative models have learned inside the deep generative representations and how photo-realistic images are able to be composed of the layer-wise stochasticity introduced in recent GANs. In this work, we show that highly-structured semantic hierarchy emerges as variation factors from synthesizing scenes from the generative representations in state-of-the-art GAN models, like StyleGAN and BigGAN. By probing the layer-wise representations with a broad set of semantics at different abstraction levels, we are able to quantify the causality between the activations and semantics occurring in the output image. Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes. The qualitative and quantitative results further suggest that the generative representations learned by the GANs with layer-wise latent codes are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme. Identifying such a set of manipulatable latent variation factors facilitates semantic scene manipulation.

15 pages, 20 figures

References in corpus (14)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• How transferable are features in deep neural networks?
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• On the importance of single directions for generalization
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Deep Photo Style Transfer
• Interpreting the Latent Space of GANs for Semantic Face Editing
• Analyzing the Performance of Multilayer Neural Networks for Object Recognition
• ImageSpirit: Verbal Guided Image Parsing
• 3D-Aware Scene Manipulation via Inverse Graphics
• ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes

Cited by in corpus (16)

• Unsupervised Discovery of Interpretable Directions in the GAN Latent Space
• Interpreting the Latent Space of GANs for Semantic Face Editing
• CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
• Visual Indeterminacy in GAN Art
• TediGAN: Text-Guided Diverse Face Image Generation and Manipulation
• GANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
• Machine learning surrogates for efficient hydrologic modeling: Insights from stochastic simulations of managed aquifer recharge
• Only a Matter of Style: Age Transformation Using a Style-Based Regression Model
• Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation
• Image Processing Using Multi-Code GAN Prior
• 2D Image Relighting with Image-to-Image Translation
• Scene relighting with illumination estimation in the latent space on an encoder-decoder scheme
• RPGAN: GANs Interpretability via Random Routing
• In-Domain GAN Inversion for Real Image Editing
• Decorating Your Own Bedroom: Locally Controlling Image Generation with Generative Adversarial Networks
• Unsupervised Image Transformation Learning via Generative Adversarial Networks