Text as Neural Operator: Image Manipulation by Text Instruction
arXiv:2008.04556 · doi:10.1145/3474085.3475343
Abstract
In recent years, text-guided image manipulation has gained increasing attention in the multimedia and computer vision community. The input to conditional image generation has evolved from image-only to multimodality. In this paper, we study a setting that allows users to edit an image with multiple objects using complex text instructions to add, remove, or change the objects. The inputs of the task are multimodal including (1) a reference image and (2) an instruction in natural language that describes desired modifications to the image. We propose a GAN-based method to tackle this problem. The key idea is to treat text as neural operators to locally modify the image feature. We show that the proposed model performs favorably against recent strong baselines on three public datasets. Specifically, it generates images of greater fidelity and semantic relevance, and when used as a image query, leads to better retrieval performance.
References in corpus (44)
- Adam: A Method for Stochastic Optimization
- Deep Residual Learning for Image Recognition
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Categorical Reparameterization with Gumbel-Softmax
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
- Toward Multimodal Image-to-Image Translation
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Pose Guided Person Image Generation
- A simple neural network module for relational reasoning
- High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
- Multimodal Unsupervised Image-to-Image Translation
- Unsupervised Attention-guided Image to Image Translation
- Semantic Image Synthesis with Spatially-Adaptive Normalization
- Multimodal Residual Learning for Visual QA
- Dialog-based Interactive Image Retrieval
- GP-GAN: Towards Realistic High-Resolution Image Blending
- Controllable Text-to-Image Generation
- Focal Visual-Text Attention for Visual Question Answering
- Learning Hierarchical Semantic Image Manipulation through Structured Representations
- Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language
- PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph
- Text-Guided Neural Image Inpainting
- A Closed-form Solution to Photorealistic Image Stylization
- Image Generation from Scene Graphs
- DeepFacePencil: Creating Face Images from Freehand Sketches
- Object-driven Text-to-Image Synthesis via Adversarial Training
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis
- Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
- Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
- FaceShop: Deep Sketch-based Face Image Editing
- Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation
- CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
- Feature Partitioning for Efficient Multi-Task Architectures
- Regularizing Generative Adversarial Networks under Limited Data
- Controllable and Progressive Image Extrapolation
- Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach
- Diverse Image Synthesis from Semantic Layouts via Conditional IMLE
- RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval
- Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
- Program-Guided Image Manipulators
- Sparsely Grouped Multi-task Generative Adversarial Networks for Facial Attribute Manipulation
- Modeling Artistic Workflows for Image Generation and Editing