Text2Scene: Generating Compositional Scenes from Textual Descriptions

Abstract

In this paper, we propose Text2Scene, a model that generates various forms of compositional scene representations from natural language descriptions. Unlike recent works, our method does NOT use Generative Adversarial Networks (GANs). Text2Scene instead learns to sequentially generate objects and their attributes (location, size, appearance, etc) at every time step by attending to different parts of the input text and the current status of the generated scene. We show that under minor modifications, the proposed framework can handle the generation of different forms of scene representations, including cartoon-like scenes, object layouts corresponding to real images, and synthetic images. Our method is not only competitive when compared with state-of-the-art GAN-based methods using automatic metrics and superior based on human judgments but also has the advantage of producing interpretable results.

arXiv commentCVPR 2019

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Generative Adversarial Text to Image Synthesis
• Learning What and Where to Draw
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• ChatPainter: Improving Text to Image Generation using Dialogue
• COCO-Stuff: Thing and Stuff Classes in Context
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication

Cited by in corpus (6)

• Generating Multiple Objects at Spatially Distinct Locations
• Interactive Image Generation Using Scene Graphs
• Image Generation from Layout
• Attribute-guided image generation from layout
• Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations
• Deep Image Synthesis from Intuitive User Input: A Review and Perspectives