StyleBank: An Explicit Representation for Neural Image Style Transfer

Abstract

We propose StyleBank, which is composed of multiple convolution filter banks and each filter bank explicitly represents one style, for neural image style transfer. To transfer an image to a specific style, the corresponding filter bank is operated on top of the intermediate feature embedding produced by a single auto-encoder. The StyleBank and the auto-encoder are jointly learnt, where the learning is conducted in such a way that the auto-encoder does not encode any style information thanks to the flexibility introduced by the explicit filter bank representation. It also enables us to conduct incremental learning to add a new image style by learning a new filter bank while holding the auto-encoder fixed. The explicit style representation along with the flexible network design enables us to fuse styles at not only the image level, but also the region level. Our method is the first style transfer network that links back to traditional texton mapping methods, and hence provides new understanding on neural style transfer. Our method is easy to train, runs in real-time, and produces results that qualitatively better or at least comparable to existing methods.

arXiv commentAccepted by CVPR 2017, corrected typos

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Texture Synthesis Using Convolutional Neural Networks
• Style-Transfer via Texture-Synthesis
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
• Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
• ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation

Cited by in corpus (44)

• Multi-style Generative Network for Real-time Transfer
• A Closed-form Solution to Photorealistic Image Stylization
• S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networks
• Coherent Online Video Style Transfer
• Gated Context Aggregation Network for Image Dehazing and Deraining
• Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN
• Visual Attribute Transfer through Deep Image Analogy
• Towards Open-Set Identity Preserving Face Synthesis
• ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
• Dynamic Instance Normalization for Arbitrary Style Transfer
• Separating Style and Content for Generalized Style Transfer
• Stereoscopic Neural Style Transfer
• Learning to Globally Edit Images with Textual Description
• Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields
• Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
• Facelet-Bank for Fast Portrait Manipulation
• Perceptual Compressive Sensing
• A Style-Aware Content Loss for Real-time HD Style Transfer
• ReCoNet: Real-time Coherent Video Style Transfer Network
• Face-to-Parameter Translation for Game Character Auto-Creation
• Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
• Rethinking and Improving the Robustness of Image Style Transfer
• Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
• Unsupervised Multi-Domain Image Translation with Domain-Specific Encoders/Decoders
• Language-Driven Image Style Transfer
• Chinese Typeface Transformation with Hierarchical Adversarial Network
• A Unified Framework for Generalizable Style Transfer: Style and Content Separation
• Parameter-Free Style Projection for Arbitrary Style Transfer
• Neural Abstract Style Transfer for Chinese Traditional Painting
• Towards Single-phase Single-stage Detection of Pulmonary Nodules in Chest CT Imaging
• Learning to Caricature via Semantic Shape Transform
• Im2Pencil: Controllable Pencil Illustration from Photographs
• Domain-Specific Mappings for Generative Adversarial Style Transfer
• Learning Selfie-Friendly Abstraction from Artistic Style Images
• Stylize Aesthetic QR Code
• Learning to Forecast Videos of Human Activity with Multi-granularity Models and Adaptive Rendering
• A Method for Arbitrary Instance Style Transfer
• Image Smoothing via Unsupervised Learning
• Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks
• The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution
• Non-Parametric Neural Style Transfer
• Learning to Transfer Visual Effects from Videos to Images
• Artistic Object Recognition by Unsupervised Style Adaptation
• TAN: Temporal Aggregation Network for Dense Multi-label Action Recognition