Image Generation from Scene Graphs

Abstract

To truly understand the visual world our models should be able not only to recognize images but also generate them. To this end, there has been exciting recent progress on generating images from natural language descriptions. These methods give stunning results on limited domains such as descriptions of birds or flowers, but struggle to faithfully reproduce complex sentences with many objects and relationships. To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships. Our model uses graph convolution to process input graphs, computes a scene layout by predicting bounding boxes and segmentation masks for objects, and converts the layout to an image with a cascaded refinement network. The network is trained adversarially against a pair of discriminators to ensure realistic outputs. We validate our approach on Visual Genome and COCO-Stuff, where qualitative results, ablations, and user studies demonstrate our method's ability to generate complex images with multiple objects.

arXiv commentTo appear at CVPR 2018

Cited by in corpus (19)

• Discrete and continuous representations and processing in deep learning: Looking forward
• Text as Neural Operator: Image Manipulation by Text Instruction
• Understanding the Role of Scene Graphs in Visual Question Answering
• Using Scene Graph Context to Improve Image Generation
• Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
• Counterfactual Critic Multi-Agent Training for Scene Graph Generation
• Dual ResGCN for Balanced Scene GraphGeneration
• SDIT: Scalable and Diverse Cross-domain Image Translation
• UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations
• Speech Fusion to Face: Bridging the Gap Between Human's Vocal Characteristics and Facial Imaging
• ORD: Object Relationship Discovery for Visual Dialogue Generation
• Learning Segmentation Masks with the Independence Prior
• Deep Consensus Learning
• Learning Predicates as Functions to Enable Few-shot Scene Graph Prediction
• Seq-SG2SL: Inferring Semantic Layout from Scene Graph Through Sequence to Sequence Learning
• Image-Graph-Image Translation via Auto-Encoding
• A Layer-Based Sequential Framework for Scene Generation with GANs
• Compact Scene Graphs for Layout Composition and Patch Retrieval
• MOC-GAN: Mixing Objects and Captions to Generate Realistic Images