Learning to Globally Edit Images with Textual Description

Abstract

We show how we can globally edit images using textual instructions: given a source image and a textual instruction for the edit, generate a new image transformed under this instruction. To tackle this novel problem, we develop three different trainable models based on RNN and Generative Adversarial Network (GAN). The models (bucket, filter bank, and end-to-end) differ in how much expert knowledge is encoded, with the most general version being purely end-to-end. To train these systems, we use Amazon Mechanical Turk to collect textual descriptions for around 2000 image pairs sampled from several datasets. Experimental results evaluated on our dataset validate our approaches. In addition, given that the filter bank model is a good compromise between generality and performance, we investigate it further by replacing RNN with Graph RNN, and show that Graph RNN improves performance. To the best of our knowledge, this is the first computational photography work on global image editing that is purely based on free-form textual instructions.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Conditional Generative Adversarial Nets
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Generative Adversarial Text to Image Synthesis
• VQA: Visual Question Answering
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Visual Storytelling
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• ChatPainter: Improving Text to Image Generation using Dialogue
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Attention Correctness in Neural Image Captioning
• ImageSpirit: Verbal Guided Image Parsing
• Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
• Neural Self Talk: Image Understanding via Continuous Questioning and Answering
• Comprehension-guided referring expressions
• Interactive Image Manipulation with Natural Language Instruction Commands

Cited by in corpus (7)

• Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach
• A Benchmark and Baseline for Language-Driven Image Editing
• Expressing Visual Relationships via Language
• Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
• Adjusting Image Attributes of Localized Regions with Low-level Dialogue
• Learning by Planning: Language-Guided Global Image Editing
• SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing