Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
arXiv:2008.01576
Abstract
We propose a novel algorithm, named Open-Edit, which is the first attempt on open-domain image manipulation with open-vocabulary instructions. It is a challenging task considering the large variation of image domains and the lack of training supervision. Our approach takes advantage of the unified visual-semantic embedding space pretrained on a general image-caption dataset, and manipulates the embedded visual features by applying text-guided vector arithmetic on the image feature maps. A structure-preserving image decoder then generates the manipulated images from the manipulated feature maps. We further propose an on-the-fly sample-specific optimization approach with cycle-consistency constraints to regularize the manipulated images and force them to preserve details of the source images. Our approach shows promising results in manipulating open-vocabulary color, texture, and high-level attributes for various scenarios of open-domain images.
arXiv commentECCV 2020. Introduction video at https://youtu.be/8E3bwvjCHYE and code at https://github.com/xh-liu/Open-Edit
References in corpus (18)
- Distributed Representations of Words and Phrases and their Compositionality
- Toward Multimodal Image-to-Image Translation
- Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
- Invertible Conditional GANs for image editing
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- Unsupervised Cross-Domain Image Generation
- Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
- Semantic Image Synthesis with Spatially-Adaptive Normalization
- Neural Photo Editing with Introspective Adversarial Networks
- GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
- Generative Visual Manipulation on the Natural Image Manifold
- InstaGAN: Instance-aware Image-to-Image Translation
- Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
- Interpreting the Latent Space of GANs for Semantic Face Editing
- Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language
- Multi-mapping Image-to-Image Translation via Learning Disentanglement
- Language Guided Fashion Image Manipulation with Feature-wise Transformations
- Bilinear Representation for Language-based Image Editing Using Conditional Generative Adversarial Networks