Generating Multiple Objects at Spatially Distinct Locations

Abstract

Recent improvements to Generative Adversarial Networks (GANs) have made it possible to generate realistic images in high resolution based on natural language descriptions such as image captions. Furthermore, conditional GANs allow us to control the image generation process through labels or even natural language descriptions. However, fine-grained control of the image layout, i.e. where in the image specific objects should be located, is still difficult to achieve. This is especially true for images that should contain multiple distinct objects at different spatial locations. We introduce a new approach which allows us to control the location of arbitrarily many objects within an image by adding an object pathway to both the generator and the discriminator. Our approach does not need a detailed semantic layout but only bounding boxes and the respective labels of the desired objects are needed. The object pathway focuses solely on the individual objects and is iteratively applied at the locations specified by the bounding boxes. The global pathway focuses on the image background and the general image layout. We perform experiments on the Multi-MNIST, CLEVR, and the more complex MS-COCO data set. Our experiments show that through the use of the object pathway we can control object locations within images and can model complex scenes with multiple objects at various locations. We further show that the object pathway focuses on the individual objects and learns features relevant for these, while the global pathway focuses on global image characteristics and the image background.

Published at ICLR 2019

References in corpus (8)

• YOLOv3: An Incremental Improvement
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Generative Adversarial Text to Image Synthesis
• Learning What and Where to Draw
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Learning Hierarchical Semantic Image Manipulation through Structured Representations
• Text2Scene: Generating Compositional Scenes from Textual Descriptions
• Deep Structured Generative Models

Cited by in corpus (11)

• Adversarial Text-to-Image Synthesis: A Review
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Investigating Object Compositionality in Generative Adversarial Networks
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Vision and Language: from Visual Perception to Content Creation
• Image Synthesis From Reconfigurable Layout and Style
• Inferring spatial relations from textual descriptions of images
• Deep Consensus Learning
• Deep Image Synthesis from Intuitive User Input: A Review and Perspectives
• Seq-SG2SL: Inferring Semantic Layout from Scene Graph Through Sequence to Sequence Learning
• Exploiting Relationship for Complex-scene Image Generation