Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration

Abstract

Relation prediction among entities in images is an important step in scene graph generation (SGG), which further impacts various visual understanding and reasoning tasks. Existing SGG frameworks, however, require heavy training yet are incapable of modeling unseen (i.e.,zero-shot) triplets. In this work, we stress that such incapability is due to the lack of commonsense reasoning,i.e., the ability to associate similar entities and infer similar relations based on general understanding of the world. To fill this gap, we propose CommOnsense-integrAted sCenegrapHrElation pRediction (COACHER), a framework to integrate commonsense knowledge for SGG, especially for zero-shot relation prediction. Specifically, we develop novel graph mining pipelines to model the neighborhoods and paths around entities in an external commonsense knowledge graph, and integrate them on top of state-of-the-art SGG frameworks. Extensive quantitative evaluations and qualitative case studies on both original and manipulated datasets from Visual Genome demonstrate the effectiveness of our proposed approach.

arXiv commentThis paper has been accepted for presentation in the Research Track of ECML-PKDD 2021

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• YOLOv3: An Incremental Improvement
• Neural Message Passing for Quantum Chemistry
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• Controllable Text-to-Image Generation
• Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
• On Support Relations and Semantic Scene Graphs
• Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks