Multi-style Generative Network for Real-time Transfer

Abstract

Despite the rapid progress in style transfer, existing approaches using feed-forward generative network for multi-style or arbitrary-style transfer are usually compromised of image quality and model flexibility. We find it is fundamentally difficult to achieve comprehensive style modeling using 1-dimensional style embedding. Motivated by this, we introduce CoMatch Layer that learns to match the second order feature statistics with the target styles. With the CoMatch Layer, we build a Multi-style Generative Network (MSG-Net), which achieves real-time performance. We also employ an specific strategy of upsampled convolution which avoids checkerboard artifacts caused by fractionally-strided convolution. Our method has achieved superior image quality comparing to state-of-the-art approaches. The proposed MSG-Net as a general approach for real-time style transfer is compatible with most existing techniques including content-style interpolation, color-preserving, spatial control and brush stroke size control. MSG-Net is the first to achieve real-time brush-size control in a purely feed-forward manner for style transfer. Our implementations and pre-trained models for Torch, PyTorch and MXNet frameworks will be publicly available.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Residual Learning for Image Recognition
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Fully Convolutional Networks for Semantic Segmentation
• A Learned Representation For Artistic Style
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Texture Synthesis Using Convolutional Neural Networks
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Mode Regularized Generative Adversarial Networks
• Fast Patch-based Style Transfer of Arbitrary Style
• Preserving Color in Neural Artistic Style Transfer
• Neural Style Transfer: A Review
• Understanding Deep Image Representations by Inverting Them
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Stacked Generative Adversarial Networks

Cited by in corpus (37)

• A Survey of Recent Advances in CNN-based Single Image Crowd Counting and Density Estimation
• Infrared and Visible Image Fusion with ResNet and zero-phase component analysis
• Context Encoding for Semantic Segmentation
• Neural Style Transfer: A Review
• Density-aware Single Image De-raining using a Multi-stream Dense Network
• Deblurring Face Images using Uncertainty Guided Multi-Stream Semantic Networks
• Confidence Measure Guided Single Image De-raining
• Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network
• Generative Adversarial Network-based Synthesis of Visible Faces from Polarimetric Thermal Faces
• Spatially Controllable Image Synthesis with Internal Representation Collaging
• Generation of microbial colonies dataset with deep learning style transfer
• TextureGAN: Controlling Deep Image Synthesis with Texture Patches
• ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
• Meta Networks for Neural Style Transfer
• Dynamic Instance Normalization for Arbitrary Style Transfer
• Separating Style and Content for Generalized Style Transfer
• Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields
• Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
• Real-time Universal Style Transfer on High-resolution Images via Zero-channel Pruning
• CoCoPIE: Making Mobile AI Sweet As PIE --Compression-Compilation Co-Design Goes a Long Way
• Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
• Deep Cross-Modal Audio-Visual Generation
• Language-Driven Image Style Transfer
• DRB-GAN: A Dynamic ResBlock Generative Adversarial Network for Artistic Style Transfer
• Domain-Adaptive Few-Shot Learning
• Block Shuffle: A Method for High-resolution Fast Style Transfer with Limited Memory
• Pixel and Feature Level Based Domain Adaption for Object Detection in Autonomous Driving
• Neural Style Transfer for Point Clouds
• Webpage Saliency Prediction with Two-stage Generative Adversarial Networks
• MetaStyle: Three-Way Trade-Off Among Speed, Flexibility, and Quality in Neural Style Transfer
• Network Architecture Search for Face Enhancement
• Image Style Transfer and Content-Style Disentanglement
• Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks
• Autonomy, Authenticity, Authorship and Intention in computer generated art
• Learning to Transfer Visual Effects from Videos to Images
• Towards Real-Time DNN Inference on Mobile Platforms with Model Pruning and Compiler Optimization
• GPU-Accelerated Mobile Multi-view Style Transfer