Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

Abstract

We propose a novel hierarchical approach for text-to-image synthesis by inferring semantic layout. Instead of learning a direct mapping from text to image, our algorithm decomposes the generation process into multiple steps, in which it first constructs a semantic layout from the text by the layout generator and converts the layout to an image by the image generator. The proposed layout generator progressively constructs a semantic layout in a coarse-to-fine manner by generating object bounding boxes and refining each box by estimating object shapes inside the box. The image generator synthesizes an image conditioned on the inferred semantic layout, which provides a useful semantic structure of an image matching with the text description. Our model not only generates semantically more meaningful images, but also allows automatic annotation of generated images and user-controlled generation process by modifying the generated scene layout. We demonstrate the capability of the proposed model on challenging MS-COCO dataset and show that the model can substantially improve the image quality, interpretability of output and semantic alignment to input text over existing approaches.

arXiv commentCVPR 2018

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Conditional Generative Adversarial Nets
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Improved Techniques for Training GANs
• Rethinking the Inception Architecture for Computer Vision
• Perceptual Adversarial Networks for Image-to-Image Transformation
• Show and Tell: A Neural Image Caption Generator
• Learning to Generate Long-term Future via Hierarchical Prediction
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Generating Images from Captions with Attention
• CIDEr: Consensus-based Image Description Evaluation
• Parallel Multiscale Autoregressive Density Estimation
• I2T2I: Learning Text to Image Synthesis with Textual Data Augmentation
• Adversarial nets with perceptual losses for text-to-image synthesis

Cited by in corpus (8)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• ChatPainter: Improving Text to Image Generation using Dialogue
• PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Learning to Globally Edit Images with Textual Description
• DwNet: Dense warp-based network for pose-guided human video generation
• LUCSS: Language-based User-customized Colourization of Scene Sketches
• MOC-GAN: Mixing Objects and Captions to Generate Realistic Images