SESAME: Semantic Editing of Scenes by Adding, Manipulating or Erasing Objects

Abstract

Recent advances in image generation gave rise to powerful tools for semantic image editing. However, existing approaches can either operate on a single image or require an abundance of additional information. They are not capable of handling the complete set of editing operations, that is addition, manipulation or removal of semantic concepts. To address these limitations, we propose SESAME, a novel generator-discriminator pair for Semantic Editing of Scenes by Adding, Manipulating or Erasing objects. In our setup, the user provides the semantic labels of the areas to be edited and the generator synthesizes the corresponding pixels. In contrast to previous methods that employ a discriminator that trivially concatenates semantics and image as an input, the SESAME discriminator is composed of two input streams that independently process the image and its semantics, using the latter to manipulate the results of the former. We evaluate our model on a diverse set of datasets and report state-of-the-art performance on two tasks: (a) image manipulation and (b) image generation conditioned on semantic labels.

References in corpus (25)

• Adam: A Method for Stochastic Optimization
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
• Improved Techniques for Training GANs
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Data Augmentation Generative Adversarial Networks
• EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
• cGANs with Projection Discriminator
• Semantic Photo Manipulation with a Generative Image Prior
• Semantic Understanding of Scenes through the ADE20K Dataset
• Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
• Pedestrian-Synthesis-GAN: Generating Pedestrian Data in Real Scene and Beyond
• Learning Hierarchical Semantic Image Manipulation through Structured Representations
• Context-Aware Synthesis and Placement of Object Instances
• Geometric GAN
• Adversarial Scene Editing: Automatic Object Removal from Weak Supervision
• FaceShop: Deep Sketch-based Face Image Editing

Cited by in corpus (9)

• You Only Need Adversarial Supervision for Semantic Image Synthesis
• Deep Learning Technique for Human Parsing: A Survey and Outlook
• Layout-to-Image Translation with Double Pooling Generative Adversarial Networks
• Image Inpainting Guided by Coherence Priors of Semantics and Textures
• Bi-level Feature Alignment for Versatile Image Translation and Manipulation
• Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis
• Semantic Editing On Segmentation Map Via Multi-Expansion Loss
• Realistic Image Synthesis with Configurable 3D Scene Layouts
• MixSyn: Learning Composition and Style for Multi-Source Image Synthesis