Sketch-R2CNN: An Attentive Network for Vector Sketch Recognition

Abstract

Freehand sketching is a dynamic process where points are sequentially sampled and grouped as strokes for sketch acquisition on electronic devices. To recognize a sketched object, most existing methods discard such important temporal ordering and grouping information from human and simply rasterize sketches into binary images for classification. In this paper, we propose a novel single-branch attentive network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the dynamics in sketches for recognition. Sketch-R2CNN takes as input only a vector sketch with grouped sequences of points, and uses an RNN for stroke attention estimation in the vector space and a CNN for 2D feature extraction in the pixel space respectively. To bridge the gap between these two spaces in neural networks, we propose a neural line rasterization module to convert the vector sketch along with the attention estimated by RNN into a bitmap image, which is subsequently consumed by CNN. The neural line rasterization module is designed in a differentiable way to yield a unified pipeline for end-to-end learning. We perform experiments on existing large-scale sketch recognition benchmarks and show that by exploiting the sketch dynamics with the attention mechanism, our method is more robust and achieves better performance than the state-of-the-art methods.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Recurrent Models of Visual Attention
• A Neural Representation of Sketch Drawings
• Feature Pyramid Networks for Object Detection
• Sketch-based 3D Shape Retrieval using Convolutional Neural Networks
• Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
• Dual Attention Networks for Multimodal Reasoning and Matching
• Sketch-a-Net that Beats Humans
• SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval
• Learning to Sketch with Shortcut Cycle Consistency
• SketchyScene: Richly-Annotated Scene Sketches
• Fast Sketch Segmentation and Labeling with Deep Learning

Cited by in corpus (4)

• Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt
• End-to-End Learning Local Multi-view Descriptors for 3D Point Clouds
• Deep Self-Supervised Representation Learning for Free-Hand Sketch
• SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence