A Style-Aware Content Loss for Real-time HD Style Transfer

Abstract

Recently, style transfer has received a lot of attention. While much of this research has aimed at speeding up processing, the approaches are still lacking from a principled, art historical standpoint: a style is more than just a single image or an artist, but previous work is limited to only a single instance of a style or shows no benefit from more images. Moreover, previous work has relied on a direct comparison of art in the domain of RGB images or on CNNs pre-trained on ImageNet, which requires millions of labeled object bounding boxes and can introduce an extra bias, since it has been assembled without artistic consideration. To circumvent these issues, we propose a style-aware content loss, which is trained jointly with a deep encoder-decoder network for real-time, high-resolution stylization of images and videos. We propose a quantitative measure for evaluating the quality of a stylized image and also have art historians rank patches from our approach against those from previous work. These and our qualitative results ranging from small image patches to megapixel stylistic images and videos show that our approach better captures the subtle nature in which a style affects content.

arXiv commentAccepted at ECCV18 (Oral)

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Generative Adversarial Networks: An Overview
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Instance Normalization: The Missing Ingredient for Fast Stylization
• A Learned Representation For Artistic Style
• SegAN: Adversarial Network with Multi-scale $L_1$ Loss for Medical Image Segmentation
• Texture Synthesis Using Convolutional Neural Networks
• Mode Regularized Generative Adversarial Networks
• Fast Patch-based Style Transfer of Arbitrary Style
• Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses
• Deep Photo Style Transfer
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Visual Attribute Transfer through Deep Image Analogy
• ZM-Net: Real-time Zero-shot Image Manipulation Network
• Meta Networks for Neural Style Transfer
• Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields

Cited by in corpus (5)

• Language-Driven Image Style Transfer
• EFANet: Exchangeable Feature Alignment Network for Arbitrary Style Transfer
• Im2Pencil: Controllable Pencil Illustration from Photographs
• Recovering Geometric Information with Learned Texture Perturbations
• CartoonRenderer: An Instance-based Multi-Style Cartoon Image Translator