Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network

Abstract

This paper presents a novel method to deal with the challenging task of generating photographic images conditioned on semantic image descriptions. Our method introduces accompanying hierarchical-nested adversarial objectives inside the network hierarchies, which regularize mid-level representations and assist generator training to capture the complex image statistics. We present an extensile single-stream generator architecture to better adapt the jointed discriminators and push generated images up to high resolutions. We adopt a multi-purpose adversarial loss to encourage more effective image and text information usage in order to improve the semantic consistency and image fidelity simultaneously. Furthermore, we introduce a new visual-semantic similarity measure to evaluate the semantic consistency of generated images. With extensive experimental validation on three public datasets, our method significantly improves previous state of the arts on all datasets over different evaluation metrics.

arXiv commentCVPR2018 Spotlight

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Pixel Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Going Deeper with Convolutions
• Improved Techniques for Training GANs
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Fully Convolutional Networks for Semantic Segmentation
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Pyramid Scene Parsing Network
• Least Squares Generative Adversarial Networks
• Dual Discriminator Generative Adversarial Nets
• Multi-style Generative Network for Real-time Transfer
• Towards Adversarial Retinal Image Synthesis
• MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
• Stacked Generative Adversarial Networks
• Semantic Image Synthesis via Adversarial Learning
• Adversarial nets with perceptual losses for text-to-image synthesis

Cited by in corpus (14)

• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Fashion-Gen: The Generative Fashion Dataset and Challenge
• Densely Connected Pyramid Dehazing Network
• MirrorGAN: Learning Text-to-image Generation by Redescription
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Consistency Regularization for Generative Adversarial Networks
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks
• Semantics Disentangling for Text-to-Image Generation
• Learning to Forecast and Refine Residual Motion for Image-to-Video Generation
• SegAttnGAN: Text to Image Generation with Segmentation Attention
• C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesis
• Bi-Semantic Reconstructing Generative Network for Zero-shot Learning
• MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination