PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph

Abstract

Despite some exciting progress on high-quality image generation from structured(scene graphs) or free-form(sentences) descriptions, most of them only guarantee the image-level semantical consistency, i.e. the generated image matching the semantic meaning of the description. They still lack the investigations on synthesizing the images in a more controllable way, like finely manipulating the visual appearance of every object. Therefore, to generate the images with preferred objects and rich interactions, we propose a semi-parametric method, PasteGAN, for generating the image from the scene graph and the image crops, where spatial arrangements of the objects and their pair-wise relationships are defined by the scene graph and the object appearances are determined by the given object crops. To enhance the interactions of the objects in the output, we design a Crop Refining Network and an Object-Image Fuser to embed the objects as well as their relationships into one map. Multiple losses work collaboratively to guarantee the generated images highly respecting the crops and complying with the scene graphs while maintaining excellent image quality. A crop selector is also proposed to pick the most-compatible crops from our external object tank by encoding the interactions around the objects in the scene graph if the crops are not provided. Evaluated on Visual Genome and COCO-Stuff dataset, our proposed method significantly outperforms the SOTA methods on Inception Score, Diversity Score and Fréchet Inception Distance. Extensive experiments also demonstrate our method's ability to generate complex and diverse images with given objects.

arXiv comment10 pages, 6 figures; Accepted by NeurIPS 2019

References in corpus (11)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Pixel Recurrent Neural Networks
• Deep Convolutional Networks on Graph-Structured Data
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Pixels to Graphs by Associative Embedding
• Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
• Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
• Interactive Image Generation Using Scene Graphs
• Disentangling Pose from Appearance in Monochrome Hand Images

Cited by in corpus (8)

• A Comprehensive Survey of Scene Graphs: Generation and Application
• Adversarial Text-to-Image Synthesis: A Review
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• ALR-GAN: Adaptive Layout Refinement for Text-to-Image Synthesis
• Text as Neural Operator: Image Manipulation by Text Instruction
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Visual-Relation Conscious Image Generation from Structured-Text
• Target-Tailored Source-Transformation for Scene Graph Generation