Controllable Text-to-Image Generation

Abstract

In this paper, we propose a novel controllable text-to-image generative adversarial network (ControlGAN), which can effectively synthesise high-quality images and also control parts of the image generation according to natural language descriptions. To achieve this, we introduce a word-level spatial and channel-wise attention-driven generator that can disentangle different visual attributes, and allow the model to focus on generating and manipulating subregions corresponding to the most relevant words. Also, a word-level discriminator is proposed to provide fine-grained supervisory feedback by correlating words with image regions, facilitating training an effective generator which is able to manipulate specific visual attributes without affecting the generation of other content. Furthermore, perceptual loss is adopted to reduce the randomness involved in the image generation, and to encourage the generator to manipulate specific attributes required in the modified text. Extensive experiments on benchmark datasets demonstrate that our method outperforms existing state of the art, and is able to effectively manipulate synthetic images using natural language descriptions. Code is available at https://github.com/mrlibw/ControlGAN.

NeurIPS 2019

Cited by in corpus (24)

• Adversarial Text-to-Image Synthesis: A Review
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse Text-to-Image Generation
• TediGAN: Text-Guided Diverse Face Image Generation and Manipulation
• Convolutional Generation of Textured 3D Meshes
• Text as Neural Operator: Image Manipulation by Text Instruction
• Improving Text-to-Image Synthesis Using Contrastive Learning
• X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
• Local Class-Specific and Global Image-Level Generative Adversarial Networks for Semantic-Guided Scene Generation
• Lightweight Generative Adversarial Networks for Text-Guided Image Manipulation
• Continuous Conditional Generative Adversarial Networks: Novel Empirical Losses and Label Input Mechanisms
• Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation
• ManiGAN: Text-Guided Image Manipulation
• CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis
• Dual Attention GANs for Semantic Image Synthesis
• DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation
• Cycle-Consistent Inverse GAN for Text-to-Image Synthesis
• MPG: A Multi-ingredient Pizza Image Generator with Conditional StyleGANs
• Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration
• Semantic Text-to-Face GAN -ST^2FG
• Adversarial Synthesis of Human Pose from Text
• Static and Animated 3D Scene Generation from Free-form Text Descriptions
• Multi-Tailed, Multi-Headed, Spatial Dynamic Memory refined Text-to-Image Synthesis