Multi-Graph Transformer for Free-Hand Sketch Recognition

Abstract

Learning meaningful representations of free-hand sketches remains a challenging task given the signal sparsity and the high-level abstraction of sketches. Existing techniques have focused on exploiting either the static nature of sketches with Convolutional Neural Networks (CNNs) or the temporal sequential property with Recurrent Neural Networks (RNNs). In this work, we propose a new representation of sketches as multiple sparsely connected graphs. We design a novel Graph Neural Network (GNN), the Multi-Graph Transformer (MGT), for learning representations of sketches from multiple graphs which simultaneously capture global and local geometric stroke structures, as well as temporal information. We report extensive numerical experiments on a sketch recognition task to demonstrate the performance of the proposed approach. Particularly, MGT applied on 414k sketches from Google QuickDraw: (i) achieves small recognition gap to the CNN-based performance upper bound (72.80% vs. 74.22%), and (ii) outperforms all RNN-based models by a significant margin. To the best of our knowledge, this is the first work proposing to represent sketches as graphs and apply GNNs for sketch recognition. Code and trained models are available at https://github.com/PengBoXiangShang/multigraph_transformer.

arXiv commentThis paper has been accepted by IEEE TNNLS

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Inductive Representation Learning on Large Graphs
• On the difficulty of training Recurrent Neural Networks
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• A Neural Representation of Sketch Drawings
• Understanding Back-Translation at Scale
• Learning Deep Transformer Models for Machine Translation
• BP-Transformer: Modelling Long-Range Context via Binary Partitioning
• Sketch-a-Net that Beats Humans
• Deep Learning for Free-Hand Sketch: A Survey
• Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers
• Enabling My Robot To Play Pictionary : Recurrent Neural Networks For Sketch Recognition
• Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition
• Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt

Cited by in corpus (10)

• The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification
• A Generalization of Transformer Networks to Graphs
• A Parallel Down-Up Fusion Network for Salient Object Detection in Optical Remote Sensing Images
• Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
• Graph-Aware Transformer: Is Attention All Graphs Need?
• Deep Self-Supervised Representation Learning for Free-Hand Sketch
• On Learning Semantic Representations for Million-Scale Free-Hand Sketches
• C-DLinkNet: considering multi-level semantic features for human parsing
• An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information
• InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping