Neural Painters: A learned differentiable constraint for generating brushstroke paintings

Abstract

We explore neural painters, a generative model for brushstrokes learned from a real non-differentiable and non-deterministic painting program. We show that when training an agent to "paint" images using brushstrokes, using a differentiable neural painter leads to much faster convergence. We propose a method for encouraging this agent to follow human-like strokes when reconstructing digits. We also explore the use of a neural painter as a differentiable image parameterization. By directly optimizing brushstrokes to activate neurons in a pre-trained convolutional network, we can directly visualize ImageNet categories and generate "ideal" paintings of each class. Finally, we present a new concept called intrinsic style transfer. By minimizing only the content loss from neural style transfer, we allow the artistic medium, in this case, brushstrokes, to naturally dictate the resulting style.

Added more references and acknowledgments

References in corpus (7)

• Conditional Generative Adversarial Nets
• Recurrent World Models Facilitate Policy Evolution
• CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms
• Synthesizing Programs for Images using Reinforced Adversarial Learning
• Neural Style Transfer: A Review
• Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting
• Unsupervised Image to Sequence Translation with Canvas-Drawer Networks

Cited by in corpus (14)

• Why Do Line Drawings Work? A Realism Hypothesis
• Unsupervised Doodling and Painting with Improved SPIRAL
• Text and Style Conditioned GAN for Generation of Offline Handwriting Lines
• KaoKore: A Pre-modern Japanese Art Facial Expression Dataset
• Differentiable Drawing and Sketching
• Generative Adversarial Networks are Special Cases of Artificial Curiosity (1990) and also Closely Related to Predictability Minimization (1991)
• Interactive Neural Style Transfer with Artists
• Generating Novel Glyph without Human Data by Learning to Communicate
• Stylized Neural Painting
• Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings
• The power of pictures: using ML assisted image generation to engage the crowd in complex socioscientific problems
• SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks
• Channel Decomposition into Painting Actions
• The Joy of Neural Painting