Bridging Knowledge Graphs to Generate Scene Graphs

Abstract

Scene graphs are powerful representations that parse images into their abstract semantic elements, i.e., objects and their interactions, which facilitates visual comprehension and explainable reasoning. On the other hand, commonsense knowledge graphs are rich repositories that encode how the world is structured, and how general concepts interact. In this paper, we present a unified formulation of these two constructs, where a scene graph is seen as an image-conditioned instantiation of a commonsense knowledge graph. Based on this new perspective, we re-formulate scene graph generation as the inference of a bridge between the scene and commonsense graphs, where each entity or predicate instance in the scene graph has to be linked to its corresponding entity or predicate class in the commonsense graph. To this end, we propose a novel graph-based neural network that iteratively propagates information between the two graphs, as well as within each of them, while gradually refining their bridge in each iteration. Our Graph Bridging Network, GB-Net, successively infers edges and nodes, allowing to simultaneously exploit and refine the rich, heterogeneous structure of the interconnected scene and commonsense graphs. Through extensive experimentation, we showcase the superior accuracy of GB-Net compared to the most recent methods, resulting in a new state of the art. We publicly release the source code of our method.

arXiv commentTo be presented at ECCV 2020

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Semi-Supervised Classification with Graph Convolutional Networks
• Gated Graph Sequence Neural Networks
• Pixels to Graphs by Associative Embedding
• Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
• Question Answering as Global Reasoning over Semantic Abstractions
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• The More You Know: Using Knowledge Graphs for Image Classification
• Learning by Abstraction: The Neural State Machine
• Modeling Text with Graph Convolutional Network for Cross-Modal Information Retrieval
• Consolidating Commonsense Knowledge

Cited by in corpus (5)

• Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
• Consolidating Commonsense Knowledge
• Type-augmented Relation Prediction in Knowledge Graphs
• Recovering the Unbiased Scene Graphs from the Biased Ones
• Fully Convolutional Scene Graph Generation