Object-driven Text-to-Image Synthesis via Adversarial Training

Abstract

In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes. Following the two-step (layout-image) generation process, a novel object-driven attentive image generator is proposed to synthesize salient objects by paying attention to the most relevant words in the text description and the pre-generated semantic layout. In addition, a new Fast R-CNN based object-wise discriminator is proposed to provide rich object-wise discrimination signals on whether the synthesized object matches the text description and the pre-generated layout. The proposed Obj-GAN significantly outperforms the previous state of the art in various metrics on the large-scale COCO benchmark, increasing the Inception score by 27% and decreasing the FID score by 11%. A thorough comparison between the traditional grid attention and the new object-driven attention is provided through analyzing their mechanisms and visualizing their attention layers, showing insights of how the proposed model generates complex scenes in high quality.

CVPR 2019

References in corpus (6)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Generative Adversarial Text to Image Synthesis
• Learning What and Where to Draw
• Turbo Learning for Captionbot and Drawingbot

Cited by in corpus (7)

• Text as Neural Operator: Image Manipulation by Text Instruction
• Text-to-Face Generation with StyleGAN2
• Image Synthesis From Reconfigurable Layout and Style
• SegAttnGAN: Text to Image Generation with Segmentation Attention
• Inferring spatial relations from textual descriptions of images
• TreeGAN: Incorporating Class Hierarchy into Image Generation
• MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination