Interactive Image Generation Using Scene Graphs
arXiv:1905.03743
Abstract
Recent years have witnessed some exciting developments in the domain of generating images from scene-based text descriptions. These approaches have primarily focused on generating images from a static text description and are limited to generating images in a single pass. They are unable to generate an image interactively based on an incrementally additive text description (something that is more intuitive and similar to the way we describe an image). We propose a method to generate an image incrementally based on a sequence of graphs of scene descriptions (scene-graphs). We propose a recurrent network architecture that preserves the image content generated in previous steps and modifies the cumulative image as per the newly provided scene information. Our model utilizes Graph Convolutional Networks (GCN) to cater to variable-sized scene graphs along with Generative Adversarial image translation networks to generate realistic multi-object images without needing any intermediate supervision during training. We experiment with Coco-Stuff dataset which has multi-object images along with annotations describing the visual scene and show that our model significantly outperforms other approaches on the same dataset in generating visually consistent images for incrementally growing scene graphs.
Published at ICLR 2019 Deep Generative Models for Highly Structured Data Workshop
References in corpus (7)
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
- An Introduction to Image Synthesis with Generative Adversarial Nets
- COCO-Stuff: Thing and Stuff Classes in Context
- Text2Scene: Generating Compositional Scenes from Textual Descriptions
- Generating Image Sequence from Description with LSTM Conditional GAN
Cited by in corpus (6)
- A Comprehensive Survey of Scene Graphs: Generation and Application
- Adversarial Text-to-Image Synthesis: A Review
- Semantic Object Accuracy for Generative Text-to-Image Synthesis
- PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph
- Triplet-Aware Scene Graph Embeddings
- Semantic Editing On Segmentation Map Via Multi-Expansion Loss