Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language

Abstract

This paper addresses the problem of manipulating images using natural language description. Our task aims to semantically modify visual attributes of an object in an image according to the text describing the new visual appearance. Although existing methods synthesize images having new attributes, they do not fully preserve text-irrelevant contents of the original image. In this paper, we propose the text-adaptive generative adversarial network (TAGAN) to generate semantically manipulated images while preserving text-irrelevant contents. The key to our method is the text-adaptive discriminator that creates word-level local discriminators according to input text to classify fine-grained attributes independently. With this discriminator, the generator learns to generate images where only regions that correspond to the given text are modified. Experimental results show that our method outperforms existing methods on CUB and Oxford-102 datasets, and our results were mostly preferred on a user study. Extensive analysis shows that our method is able to effectively disentangle visual attributes and produce pleasing outputs.

arXiv commentNeurIPS 2018

Cited by in corpus (20)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Adversarial Text-to-Image Synthesis: A Review
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Text as Neural Operator: Image Manipulation by Text Instruction
• Towards Open-World Text-Guided Face Image Generation and Manipulation
• Image-to-Image Translation with Text Guidance
• Lightweight Generative Adversarial Networks for Text-Guided Image Manipulation
• Language-Driven Image Style Transfer
• Sequential Attention GAN for Interactive Image Editing
• Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
• Remember What You have drawn: Semantic Image Manipulation with Memory
• A Benchmark and Baseline for Language-Driven Image Editing
• I Want This Product but Different : Multimodal Retrieval with Synthetic Query Expansion
• Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model
• Attribute-guided image generation from layout
• Grounded and Controllable Image Completion by Incorporating Lexical Semantics
• Semantic Text-to-Face GAN -ST^2FG
• SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing
• Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling