Language-Driven Image Style Transfer

Abstract

Despite having promising results, style transfer, which requires preparing style images in advance, may result in lack of creativity and accessibility. Following human instruction, on the other hand, is the most natural way to perform artistic style transfer that can significantly improve controllability for visual effect applications. We introduce a new task, language-driven artistic style transfer (LDAST), to manipulate the style of a content image, guided by a text. We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDAST by the patch-wise style discriminator. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. CLVA further compares contrastive pairs of content images and style instructions to improve the mutual relativeness. The results from the same content image can preserve consistent content structures. Besides, they should present analogous style patterns from style instructions that contain similar visual semantics. The experiments show that our CLVA is effective and achieves superb transferred results on LDAST.

ECCV'22

References in corpus (18)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Dual Learning for Machine Translation
• Fast Patch-based Style Transfer of Arbitrary Style
• Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses
• Contrastive Learning for Unpaired Image-to-Image Translation
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• MirrorGAN: Learning Text-to-image Generation by Redescription
• Frustratingly Simple Domain Generalization via Image Stylization
• Coherent Online Video Style Transfer
• Semantic Image Synthesis via Adversarial Learning
• Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN
• Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments
• Diversified Texture Synthesis with Feed-forward Networks
• Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition
• Contrastive Visual-Linguistic Pretraining
• Image Super-Resolution by Neural Texture Transfer
• Improved Sentiment Detection via Label Transfer from Monolingual to Synthetic Code-Switched Text