AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Abstract

In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation. With a novel attentional generative network, the AttnGAN can synthesize fine-grained details at different subregions of the image by paying attentions to the relevant words in the natural language description. In addition, a deep attentional multimodal similarity model is proposed to compute a fine-grained image-text matching loss for training the generator. The proposed AttnGAN significantly outperforms the previous state of the art, boosting the best reported inception score by 14.14% on the CUB dataset and 170.25% on the more challenging COCO dataset. A detailed analysis is also performed by visualizing the attention layers of the AttnGAN. It for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.

References in corpus (11)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Improved Techniques for Training GANs
• DRAW: A Recurrent Neural Network For Image Generation
• Rethinking the Inception Architecture for Computer Vision
• Stacked Attention Networks for Image Question Answering
• From Captions to Visual Concepts and Back
• Generating Images from Captions with Attention
• Semantic Compositional Networks for Visual Captioning

Cited by in corpus (8)

• Attention Mechanisms in Computer Vision: A Survey
• SpaText: Spatio-Textual Representation for Controllable Image Generation
• ChatPainter: Improving Text to Image Generation using Dialogue
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Interactive Image Generation Using Scene Graphs
• JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets
• Class Knowledge Overlay to Visual Feature Learning for Zero-Shot Image Classification
• BLM-SGAN: Bidirectional Language Modeling for Semantic-Spatial Text-to-Image Generation