Learning Hierarchical Semantic Image Manipulation through Structured Representations

Abstract

Understanding, reasoning, and manipulating semantic concepts of images have been a fundamental research problem for decades. Previous work mainly focused on direct manipulation on natural image manifold through color strokes, key-points, textures, and holes-to-fill. In this work, we present a novel hierarchical framework for semantic image manipulation. Key to our hierarchical framework is that we employ a structured semantic layout as our intermediate representation for manipulation. Initialized with coarse-level bounding boxes, our structure generator first creates pixel-wise semantic layout capturing the object shape, object-object interactions, and object-scene relations. Then our image generator fills in the pixel-level textures guided by the semantic layout. Such framework allows a user to manipulate images at object-level by adding, removing, and moving one bounding box at a time. Experimental evaluations demonstrate the advantages of the hierarchical manipulation framework over existing image generation and context hole-filing models, both qualitatively and quantitatively. Benefits of the hierarchical framework are further demonstrated in applications such as semantic object manipulation, interactive image editing, and data-driven image manipulation.

Cited by in corpus (23)

• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Swapping Autoencoder for Deep Image Manipulation
• SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects
• Generating Multiple Objects at Spatially Distinct Locations
• Text as Neural Operator: Image Manipulation by Text Instruction
• Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss
• SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing
• Decompose to manipulate: Manipulable Object Synthesis in 3D Medical Images with Structured Image Decomposition
• MichiGAN: Multi-Input-Conditioned Hair Image Generation for Portrait Editing
• AdvSPADE: Realistic Unrestricted Attacks for Semantic Segmentation
• Fine-grained Synthesis of Unrestricted Adversarial Examples
• Image Inpainting Guided by Coherence Priors of Semantics and Textures
• PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions
• Bi-level Feature Alignment for Versatile Image Translation and Manipulation
• GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving
• Semantically Multi-modal Image Synthesis
• GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes
• Refining Semantic Segmentation with Superpixel by Transparent Initialization and Sparse Encoder
• Semantic Editing On Segmentation Map Via Multi-Expansion Loss
• Vehicle Image Generation Going Well with The Surroundings
• DeFLOCNet: Deep Image Editing via Flexible Low-level Controls
• Hidden Footprints: Learning Contextual Walkability from 3D Human Trails
• Generating Object Stamps