Demystifying Neural Style Transfer

Abstract

Neural Style Transfer has recently demonstrated very exciting results which catches eyes in both academia and industry. Despite the amazing results, the principle of neural style transfer, especially why the Gram matrices could represent style remains unclear. In this paper, we propose a novel interpretation of neural style transfer by treating it as a domain adaptation problem. Specifically, we theoretically show that matching the Gram matrices of feature maps is equivalent to minimize the Maximum Mean Discrepancy (MMD) with the second order polynomial kernel. Thus, we argue that the essence of neural style transfer is to match the feature distributions between the style images and the generated images. To further support our standpoint, we experiment with several other distribution alignment methods, and achieve appealing results. We believe this novel interpretation connects these two important research fields, and could enlighten future researches.

Accepted by IJCAI 2017

References in corpus (3)

• Learning Transferable Features with Deep Adaptation Networks
• Deep Domain Confusion: Maximizing for Domain Invariance
• Domain Adaptations for Computer Vision Applications

Cited by in corpus (55)

• Deep Visual Domain Adaptation: A Survey
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Slimmable Neural Networks
• Learning Linear Transformations for Fast Arbitrary Style Transfer
• Semantic-aware Grad-GAN for Virtual-to-Real Urban Scene Adaption
• Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach
• Image-Image Domain Adaptation with Preserved Self-Similarity and Domain-Dissimilarity for Person Re-identification
• Style Transfer in Text: Exploration and Evaluation
• ZM-Net: Real-time Zero-shot Image Manipulation Network
• Domain Generalization Using a Mixture of Multiple Latent Domains
• Style Mixer: Semantic-aware Multi-Style Transfer Network
• DCAN: Dual Channel-wise Alignment Networks for Unsupervised Scene Adaptation
• ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
• Meta Networks for Neural Style Transfer
• Dynamic Instance Normalization for Arbitrary Style Transfer
• Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields
• Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
• ACE: Adapting to Changing Environments for Semantic Segmentation
• Real-time Universal Style Transfer on High-resolution Images via Zero-channel Pruning
• Fingerprint Spoof Generalization
• A Closed-form Solution to Universal Style Transfer
• Demystifying Inter-Class Disentanglement
• Learning Style-Aware Symbolic Music Representations by Adversarial Autoencoders
• Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
• Language-Driven Image Style Transfer
• A Unified Framework for Generalizable Style Transfer: Style and Content Separation
• EDIT: Exemplar-Domain Aware Image-to-Image Translation
• ImagineNet: Restyling Apps Using Neural Style Transfer
• Channel-wise Alignment for Adaptive Object Detection
• Neural Stereoscopic Image Style Transfer
• Neural Style Transfer for Point Clouds
• Learning to Caricature via Semantic Shape Transform
• Multi-Task Learning for Automotive Foggy Scene Understanding via Domain Adaptation to an Illumination-Invariant Representation
• ThumbNet: One Thumbnail Image Contains All You Need for Recognition
• Robust Representation Learning via Perceptual Similarity Metrics
• Style is a Distribution of Features
• Exemplar-based synthesis of geology using kernel discrepancies and generative neural networks
• Low-Resource Sequence Labeling via Unsupervised Multilingual Contextualized Representations
• Uncorrelated Feature Encoding for Faster Image Style Transfer
• TET-GAN: Text Effects Transfer via Stylization and Destylization
• MetaStyle: Three-Way Trade-Off Among Speed, Flexibility, and Quality in Neural Style Transfer
• The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution
• Cross-Domain Image Classification through Neural-Style Transfer Data Augmentation
• Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization
• Using a Bi-directional LSTM Model with Attention Mechanism trained on MIDI Data for Generating Unique Music
• Demystifying Brain Tumour Segmentation Networks: Interpretability and Uncertainty Analysis
• Machine Learning in Application to Automatic Noise Processing of Solar Spectrograms
• Can We Teach Computers to Understand Art? Domain Adaptation for Enhancing Deep Networks Capacity to De-Abstract Art
• Multi-Pair Text Style Transfer on Unbalanced Data
• Global Image Sentiment Transfer
• Neural Comic Style Transfer: Case Study
• Image Synthesis and Style Transfer
• Generalize Ultrasound Image Segmentation via Instant and Plug & Play Style Transfer
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• What Can Style Transfer and Paintings Do For Model Robustness?