Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation

Abstract

We propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects. As the user starts to draw a sketch of a desired object type, the network interactively recommends plausible completions, and shows a corresponding synthesized image to the user. This enables a feedback loop, where the user can edit their sketch based on the network's recommendations, visualizing both the completed shape and final rendered image while they draw. In order to use a single trained model across a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network. Video available at our website: https://arnabgho.github.io/iSketchNFill/.

ICCV 2019, Video Avaiable at https://youtu.be/T9xtpAMUDps

References in corpus (14)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Conditional Image Generation with PixelCNN Decoders
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Which Training Methods for GANs do actually Converge?
• Adversarially Learned Inference
• A Neural Representation of Sketch Drawings
• Unrolled Generative Adversarial Networks
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• Mode Regularized Generative Adversarial Networks
• Synthesizing Programs for Images using Reinforced Adversarial Learning
• Im2Pencil: Controllable Pencil Illustration from Photographs

Cited by in corpus (4)

• Facial-Sketch Synthesis: A New Challenge
• Text as Neural Operator: Image Manipulation by Text Instruction
• One Sketch for All: One-Shot Personalized Sketch Segmentation
• Interactive Neural Painting