Learning Predicates as Functions to Enable Few-shot Scene Graph Prediction

Abstract

Scene graph prediction --- classifying the set of objects and predicates in a visual scene --- requires substantial training data. However, most predicates only occur a handful of times making them difficult to learn. We introduce the first scene graph prediction model that supports few-shot learning of predicates. Existing scene graph generation models represent objects using pretrained object detectors or word embeddings that capture semantic object information at the cost of encoding information about which relationships they afford. So, these object representations are unable to generalize to new few-shot relationships. We introduce a framework that induces object representations that are structured according to their visual relationships. Unlike past methods, our framework embeds objects that afford similar relationships closer together. This property allows our model to perform well in the few-shot setting. For example, applying the 'riding' predicate transformation to 'person' modifies the representation towards objects like 'skateboard' and 'horse' that enable riding. We generate object representations by learning predicates trained as message passing functions within a new graph convolution framework. The object representations are used to build few-shot predicate classifiers for rare predicates with as few as 1 labeled example. We achieve a 5-shot performance of 22.70 recall@50, a 3.7 increase when compared to strong transfer learning baselines.

arXiv comment14 pages, 10 figures, preprint

References in corpus (17)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Semi-Supervised Classification with Graph Convolutional Networks
• Prototypical Networks for Few-shot Learning
• Neural Message Passing for Quantum Chemistry
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Deep Convolutional Networks on Graph-Structured Data
• Dynamic Memory Networks for Visual and Textual Question Answering
• Gated Graph Sequence Neural Networks
• Few-Shot Learning with Graph Neural Networks
• Pixels to Graphs by Associative Embedding
• Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
• Modeling Relational Data with Graph Convolutional Networks
• Generative Adversarial Residual Pairwise Networks for One Shot Learning
• Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
• Image Generation from Scene Graphs
• Natural Language Guided Visual Relationship Detection
• Graph R-CNN for Scene Graph Generation

Cited by in corpus (2)

• Action Genome: Actions as Composition of Spatio-temporal Scene Graphs
• Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation