LinkNet: Relational Embedding for Scene Graph

Abstract

Objects and their relationships are critical contents for image understanding. A scene graph provides a structured description that captures these properties of an image. However, reasoning about the relationships between objects is very challenging and only a few recent works have attempted to solve the problem of generating a scene graph from an image. In this paper, we present a method that improves scene graph generation by explicitly modeling inter-dependency among the entire object instances. We design a simple and effective relational embedding module that enables our model to jointly represent connections among all related objects, rather than focus on an object in isolation. Our method significantly benefits the main part of the scene graph generation task: relationship classification. Using it on top of a basic Faster R-CNN, our model achieves state-of-the-art results on the Visual Genome benchmark. We further push the performance by introducing global context encoding module and geometrical layout encoding module. We validate our final model, LinkNet, through extensive ablation studies, demonstrating its efficacy in scene graph generation.

arXiv commentAccepted to NIPS 2018

References in corpus (13)

• Deep Convolutional Networks on Graph-Structured Data
• Learning Convolutional Neural Networks for Graphs
• A simple neural network module for relational reasoning
• Pixels to Graphs by Associative Embedding
• Visual Relationship Detection with Language Priors
• Scene Graph Generation by Iterative Message Passing
• Visual Interaction Networks
• Detecting Visual Relationships with Deep Relational Networks
• Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
• Iterative Visual Reasoning Beyond Convolutions
• Learning to Detect Human-Object Interactions
• Towards Context-aware Interaction Recognition