Unsupervised Sketch-to-Photo Synthesis
arXiv:1909.08313
Abstract
Humans can envision a realistic photo given a free-hand sketch that is not only spatially imprecise and geometrically distorted but also without colors and visual details. We study unsupervised sketch-to-photo synthesis for the first time, learning from unpaired sketch-photo data where the target photo for a sketch is unknown during training. Existing works only deal with style change or spatial deformation alone, synthesizing photos from edge-aligned line drawings or transforming shapes within the same modality, e.g., color images. Our key insight is to decompose unsupervised sketch-to-photo synthesis into a two-stage translation task: First shape translation from sketches to grayscale photos and then content enrichment from grayscale to color photos. We also incorporate a self-supervised denoising objective and an attention module to handle abstraction and style variations that are inherent and specific to sketches. Our synthesis is sketch-faithful and photo-realistic to enable sketch-based image retrieval in practice. An exciting corollary product is a universal and promising sketch generator that captures human visual perception beyond the edge map of a photo.
arXiv comment23 pages, 15 figures. URL: https://github.com/rt219/Unpaired-Sketch-to-Photo-Translation
References in corpus (6)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Conditional Generative Adversarial Nets
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- A Neural Representation of Sketch Drawings
- U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
- Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval