Stylized Neural Painting

Abstract

This paper proposes an image-to-painting translation method that generates vivid and realistic painting artworks with controllable styles. Different from previous image-to-image translation methods that formulate the translation as pixel-wise prediction, we deal with such an artistic creation process in a vectorized environment and produce a sequence of physically meaningful stroke parameters that can be further used for rendering. Since a typical vector render is not differentiable, we design a novel neural renderer which imitates the behavior of the vector renderer and then frame the stroke prediction as a parameter searching process that maximizes the similarity between the input and the rendering output. We explored the zero-gradient problem on parameter searching and propose to solve this problem from an optimal transportation perspective. We also show that previous neural renderers have a parameter coupling problem and we re-design the rendering network with a rasterization network and a shading network that better handles the disentanglement of shape and color. Experiments show that the paintings generated by our method have a high degree of fidelity in both global appearance and local textures. Our method can be also jointly optimized with neural style transfer that further transfers visual style from other images. Our code and animated results are available at \url{https://jiupinjia.github.io/neuralpainter/}.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting
• Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distance
• RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
• Learning to Paint With Model-based Deep Reinforcement Learning
• Neural Painters: A learned differentiable constraint for generating brushstroke paintings
• Learning to Sketch with Deep Q Networks and Demonstrated Strokes
• Face-to-Parameter Translation for Game Character Auto-Creation

Cited by in corpus (1)

• Convolutional Neural Network Interpretability with General Pattern Theory