Language-Driven Image Style Transfer

Abstract

Despite having promising results, style transfer, which requires preparing style images in advance, may result in lack of creativity and accessibility. Following human instruction, on the other hand, is the most natural way to perform artistic style transfer that can significantly improve controllability for visual effect applications. We introduce a new task, language-driven artistic style transfer (LDAST), to manipulate the style of a content image, guided by a text. We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDAST by the patch-wise style discriminator. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. CLVA further compares contrastive pairs of content images and style instructions to improve the mutual relativeness. The results from the same content image can preserve consistent content structures. Besides, they should present analogous style patterns from style instructions that contain similar visual semantics. The experiments show that our CLVA is effective and achieves superb transferred results on LDAST.

arXiv commentECCV'22

References in corpus (45)

• A Simple Framework for Contrastive Learning of Visual Representations
• Deep Residual Learning for Image Recognition
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Generative Adversarial Text to Image Synthesis
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Dual Learning for Machine Translation
• An efficient framework for learning sentence representations
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• Fast Patch-based Style Transfer of Arbitrary Style
• Leveraging Frequency Analysis for Deep Fake Image Recognition
• Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks
• Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Contrastive Learning for Unpaired Image-to-Image Translation
• Multi-style Generative Network for Real-time Transfer
• Learning Linear Transformations for Fast Arbitrary Style Transfer
• Demystifying Neural Style Transfer
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• MirrorGAN: Learning Text-to-image Generation by Redescription
• Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
• Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language
• ImageSpirit: Verbal Guided Image Parsing
• Frustratingly Simple Domain Generalization via Image Stylization
• A Closed-form Solution to Photorealistic Image Stylization
• Deep Learning Models for Multilingual Hate Speech Detection
• Coherent Online Video Style Transfer
• Semantic Image Synthesis via Adversarial Learning
• Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments
• Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN
• Diversified Texture Synthesis with Feed-forward Networks
• Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition
• Characterizing and Improving Stability in Neural Style Transfer
• Contrastive Visual-Linguistic Pretraining
• Stereoscopic Neural Style Transfer
• Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields
• Image Super-Resolution by Neural Texture Transfer
• A Style-Aware Content Loss for Real-time HD Style Transfer
• Interactive Image Manipulation with Natural Language Instruction Commands
• Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
• Improved Sentiment Detection via Label Transfer from Monolingual to Synthetic Code-Switched Text