Scene Graph Generation with External Knowledge and Image Reconstruction

Abstract

Scene graph generation has received growing attention with the advancements in image understanding tasks such as object detection, attributes and relationship prediction,~\etc. However, existing datasets are biased in terms of object and relationship labels, or often come with noisy and missing annotations, which makes the development of a reliable scene graph prediction model very challenging. In this paper, we propose a novel scene graph generation algorithm with external knowledge and image reconstruction loss to overcome these dataset issues. In particular, we extract commonsense knowledge from the external knowledge base to refine object and phrase features for improving generalizability in scene graph generation. To address the bias of noisy object annotations, we introduce an auxiliary image reconstruction path to regularize the scene graph generation network. Extensive experiments show that our framework can generate better scene graphs, achieving the state-of-the-art performance on two benchmark datasets: Visual Relationship Detection and Visual Genome datasets.

10 pages, 5 figures, Accepted in CVPR 2019

References in corpus (8)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Generative Adversarial Text to Image Synthesis
• Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• FVQA: Fact-based Visual Question Answering
• Natural Language Guided Visual Relationship Detection
• Image Captioning and Visual Question Answering Based on Attributes and External Knowledge

Cited by in corpus (8)

• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
• A Review on Intelligent Object Perception Methods Combining Knowledge-based Reasoning and Machine Learning
• Dual ResGCN for Balanced Scene GraphGeneration
• Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation
• Scaling Object Detection by Transferring Classification Weights
• Recovering the Unbiased Scene Graphs from the Biased Ones
• Co-embedding of Nodes and Edges with Graph Neural Networks
• MeronymNet: A Hierarchical Approach for Unified and Controllable Multi-Category Object Generation