Differentiable Drawing and Sketching

Abstract

We present a bottom-up differentiable relaxation of the process of drawing points, lines and curves into a pixel raster. Our approach arises from the observation that rasterising a pixel in an image given parameters of a primitive can be reformulated in terms of the primitive's distance transform, and then relaxed to allow the primitive's parameters to be learned. This relaxation allows end-to-end differentiable programs and deep networks to be learned and optimised and provides several building blocks that allow control over how a compositional drawing process is modelled. We emphasise the bottom-up nature of our proposed approach, which allows for drawing operations to be composed in ways that can mimic the physical reality of drawing rather than being tied to, for example, approaches in modern computer graphics. With the proposed approach we demonstrate how sketches can be generated by directly optimising against photographs and how auto-encoders can be built to transform rasterised handwritten digits into vectors without supervision. Extensive experimental results highlight the power of this approach under different modelling assumptions for drawing tasks.

References in corpus (7)

• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• A Neural Representation of Sketch Drawings
• Synthesizing Programs for Images using Reinforced Adversarial Learning
• Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting
• Neural Painters: A learned differentiable constraint for generating brushstroke paintings
• Unsupervised Doodling and Painting with Improved SPIRAL
• Teaching GANs to Sketch in Vector Format

Cited by in corpus (2)

• Learning to Draw: Emergent Communication through Sketching
• Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent?