ZM-Net: Real-time Zero-shot Image Manipulation Network

Abstract

Many problems in image processing and computer vision (e.g. colorization, style transfer) can be posed as 'manipulating' an input image into a corresponding output image given a user-specified guiding signal. A holy-grail solution towards generic image manipulation should be able to efficiently alter an input image with any personalized signals (even signals unseen during training), such as diverse paintings and arbitrary descriptive attributes. However, existing methods are either inefficient to simultaneously process multiple signals (let alone generalize to unseen signals), or unable to handle signals from other modalities. In this paper, we make the first attempt to address the zero-shot image manipulation task. We cast this problem as manipulating an input image according to a parametric model whose key parameters can be conditionally generated from any guiding signal (even unseen ones). To this end, we propose the Zero-shot Manipulation Net (ZM-Net), a fully-differentiable architecture that jointly optimizes an image-transformation network (TNet) and a parameter network (PNet). The PNet learns to generate key transformation parameters for the TNet given any guiding signal while the TNet performs fast zero-shot image manipulation according to both signal-dependent parameters from the PNet and signal-invariant parameters from the TNet itself. Extensive experiments show that our ZM-Net can perform high-quality image manipulation conditioned on different forms of guiding signals (e.g. style images and attributes) in real-time (tens of milliseconds per image) even for unseen signals. Moreover, a large-scale style dataset with over 20,000 style images is also constructed to promote further research.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Fully Convolutional Networks for Semantic Segmentation
• Learning Deconvolution Network for Semantic Segmentation
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• Demystifying Neural Style Transfer
• Diversified Texture Synthesis with Feed-forward Networks

Cited by in corpus (12)

• Multimodal Unsupervised Image-to-Image Translation
• Neural Style Transfer: A Review
• Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification
• Structured Generative Adversarial Networks
• Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
• The Intelligent ICU Pilot Study: Using Artificial Intelligence Technology for Autonomous Patient Monitoring
• A Style-Aware Content Loss for Real-time HD Style Transfer
• Arbitrary Style Transfer via Multi-Adaptation Network
• Neural Stereoscopic Image Style Transfer
• GAN Memory with No Forgetting
• StyleRemix: An Interpretable Representation for Neural Image Style Transfer
• Generating Handwriting via Decoupled Style Descriptors