Learning to Paint With Model-based Deep Reinforcement Learning

Abstract

We show how to teach machines to paint like human painters, who can use a small number of strokes to create fantastic paintings. By employing a neural renderer in model-based Deep Reinforcement Learning (DRL), our agents learn to determine the position and color of each stroke and make long-term plans to decompose texture-rich images into strokes. Experiments demonstrate that excellent visual effects can be achieved using hundreds of strokes. The training process does not require the experience of human painters or stroke tracking data. The code is available at https://github.com/hzwer/ICCV2019-LearningToPaint.

arXiv commentAccepted to ICCV 2019

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Continuous control with deep reinforcement learning
• Conditional Generative Adversarial Nets
• A Neural Representation of Sketch Drawings
• Recurrent World Models Facilitate Policy Evolution
• Synthesizing Programs for Images using Reinforced Adversarial Learning
• Investigating Human Priors for Playing Video Games
• Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories

Cited by in corpus (16)

• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Unsupervised Doodling and Painting with Improved SPIRAL
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Creative Sketch Generation
• Tuning-free Plug-and-Play Proximal Algorithm for Inverse Imaging Problems
• KaoKore: A Pre-modern Japanese Art Facial Expression Dataset
• ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control
• TFPnP: Tuning-free Plug-and-Play Proximal Algorithm with Applications to Inverse Imaging Problems
• Generative Adversarial Networks are Special Cases of Artificial Curiosity (1990) and also Closely Related to Predictability Minimization (1991)
• Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings
• PlotThread: Creating Expressive Storyline Visualizations using Reinforcement Learning
• Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale
• Stylized Neural Painting
• Using Python for Model Inference in Deep Learning
• Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning
• SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks