Sketch-based Creativity Support Tools using Deep Learning

Abstract

Sketching is a natural and effective visual communication medium commonly used in creative processes. Recent developments in deep-learning models drastically improved machines' ability in understanding and generating visual content. An exciting area of development explores deep-learning approaches used to model human sketches, opening opportunities for creative applications. This chapter describes three fundamental steps in developing deep-learning-driven creativity support tools that consumes and generates sketches: 1) a data collection effort that generated a new paired dataset between sketches and mobile user interfaces; 2) a sketch-based user interface retrieval system adapted from state-of-the-art computer vision techniques; and, 3) a conversational sketching system that supports the novel interaction of a natural-language-based sketch/critique authoring process. In this chapter, we survey relevant prior work in both the deep-learning and human-computer-interaction communities, document the data collection process and the systems' architectures in detail, present qualitative and quantitative results, and paint the landscape of several future research directions in this exciting area.

arXiv commentPreprint of chapter in published in "Artificial Intelligence for Human Computer Interaction: A Modern Approach". arXiv admin note: substantial text overlap with arXiv:2005.07781

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• A Neural Representation of Sketch Drawings
• A note on the evaluation of generative models
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Adaptively Learning the Crowd Kernel
• Conceptual Metaphors Impact Perceptions of Human-AI Collaboration
• Learning to Find Eye Region Landmarks for Remote Gaze Estimation in Unconstrained Settings
• Evorus: A Crowd-powered Conversational Assistant Built to Automate Itself Over Time
• Skill Rating for Generative Models
• Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval
• Iris: A Conversational Agent for Complex Tasks
• CoSE: Compositional Stroke Embeddings
• The DIDI dataset: Digital Ink Diagram data
• Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications
• DeepWriting: Making Digital Ink Editable via Deep Generative Modeling
• How2Sketch: Generating Easy-To-Follow Tutorials for Sketching 3D Objects