MirrorGAN: Learning Text-to-image Generation by Redescription

Abstract

Generating an image from a given text description has two goals: visual realism and semantic consistency. Although significant progress has been made in generating high-quality and visually realistic images using generative adversarial networks, guaranteeing semantic consistency between the text description and visual content remains very challenging. In this paper, we address this problem by proposing a novel global-local attentive and semantic-preserving text-to-image-to-text framework called MirrorGAN. MirrorGAN exploits the idea of learning text-to-image generation by redescription and consists of three modules: a semantic text embedding module (STEM), a global-local collaborative attentive module for cascaded image generation (GLAM), and a semantic text regeneration and alignment module (STREAM). STEM generates word- and sentence-level embeddings. GLAM has a cascaded architecture for generating target images from coarse to fine scales, leveraging both local word attention and global sentence attention to progressively enhance the diversity and semantic consistency of the generated images. STREAM seeks to regenerate the text description from the generated image, which semantically aligns with the given text description. Thorough experiments on two public benchmark datasets demonstrate the superiority of MirrorGAN over other representative state-of-the-art methods.

Accepted by CVPR 2019

References in corpus (5)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Learning What and Where to Draw
• Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network
• Exploring Human-like Attention Supervision in Visual Question Answering

Cited by in corpus (10)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis
• Language-Driven Image Style Transfer
• Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings
• Continuous Conditional Generative Adversarial Networks (cGAN) with Generator Regularization
• Conditional Image Generation and Manipulation for User-Specified Content
• TreeGAN: Incorporating Class Hierarchy into Image Generation
• MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination