Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses

Abstract

Recently, methods have been proposed that perform texture synthesis and style transfer by using convolutional neural networks (e.g. Gatys et al. [2015,2016]). These methods are exciting because they can in some cases create results with state-of-the-art quality. However, in this paper, we show these methods also have limitations in texture quality, stability, requisite parameter tuning, and lack of user controls. This paper presents a multiscale synthesis pipeline based on convolutional neural networks that ameliorates these issues. We first give a mathematical explanation of the source of instabilities in many previous approaches. We then improve these instabilities by using histogram losses to synthesize textures that better statistically match the exemplar. We also show how to integrate localized style losses in our multiscale framework. These losses can improve the quality of large features, improve the separation of content and style, and offer artistic controls such as paint by numbers. We demonstrate that our approach offers improved quality, convergence in fewer iterations, and more stability over the optimization.

References in corpus (3)

• Texture Synthesis Using Convolutional Neural Networks
• Fast Patch-based Style Transfer of Arbitrary Style
• Incorporating long-range consistency in CNN-based texture generation

Cited by in corpus (48)

• A survey of synthetic data augmentation methods in computer vision
• Neural Style Transfer: A Review
• Learning Linear Transformations for Fast Arbitrary Style Transfer
• Transport-Based Neural Style Transfer for Smoke Simulations
• A Bayesian Inference Framework for Procedural Material Parameter Estimation
• On Demand Solid Texture Synthesis Using Deep 3D Networks
• Style Mixer: Semantic-aware Multi-Style Transfer Network
• Quality Metric Guided Portrait Line Drawing Generation from Unpaired Training Data
• ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
• Meta Networks for Neural Style Transfer
• Separating Style and Content for Generalized Style Transfer
• Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
• Real-time Universal Style Transfer on High-resolution Images via Zero-channel Pruning
• A Style-Aware Content Loss for Real-time HD Style Transfer
• Controlling Perceptual Factors in Neural Style Transfer
• Drafting and Revision: Laplacian Pyramid Network for Fast High-Quality Artistic Style Transfer
• Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
• Quantitative Evaluation of Style Transfer
• A Closed-form Solution to Universal Style Transfer
• Face-to-Parameter Translation for Game Character Auto-Creation
• NITES: A Non-Parametric Interpretable Texture Synthesis Method
• Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks
• Recovering Faces from Portraits with Auxiliary Facial Attributes
• Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
• Domain Adaptation Using Adversarial Learning for Autonomous Navigation
• Collaborative Distillation for Ultra-Resolution Universal Style Transfer
• Language-Driven Image Style Transfer
• Deep Line Art Video Colorization with a Few References
• A Unified Framework for Generalizable Style Transfer: Style and Content Separation
• EDIT: Exemplar-Domain Aware Image-to-Image Translation
• Disentangling Structure and Appearance in ViT Feature Space
• TGHop: An Explainable, Efficient and Lightweight Method for Texture Generation
• FaceController: Controllable Attribute Editing for Face in the Wild
• Fast Universal Style Transfer for Artistic and Photorealistic Rendering
• EFANet: Exchangeable Feature Alignment Network for Arbitrary Style Transfer
• Style-Aware Normalized Loss for Improving Arbitrary Style Transfer
• Neural Smoke Stylization with Color Transfer
• Interactive Neural Style Transfer with Artists
• Style is a Distribution of Features
• Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport
• Defective samples simulation through Neural Style Transfer for automatic surface defect segment
• GANILLA: Generative Adversarial Networks for Image to Illustration Translation
• Non-Parametric Neural Style Transfer
• Trace-Norm Adversarial Examples
• Neural Style Difference Transfer and Its Application to Font Generation
• A Content Transformation Block For Image Style Transfer
• Global Image Sentiment Transfer
• CAMS: Color-Aware Multi-Style Transfer