Attentive Relational Networks for Mapping Images to Scene Graphs
arXiv:1811.10696
Abstract
Scene graph generation refers to the task of automatically mapping an image into a semantic structural graph, which requires correctly labeling each extracted object and their interaction relationships. Despite the recent success in object detection using deep learning techniques, inferring complex contextual relationships and structured graph representations from visual data remains a challenging topic. In this study, we propose a novel Attentive Relational Network that consists of two key modules with an object detection backbone to approach this problem. The first module is a semantic transformation module utilized to capture semantic embedded relation features, by translating visual features and linguistic features into a common semantic space. The other module is a graph self-attention module introduced to embed a joint graph representation through assigning various importance weights to neighboring nodes. Finally, accurate scene graphs are produced by the relation inference module to recognize all entities and the corresponding relations. We evaluate our proposed method on the widely-adopted Visual Genome Dataset, and the results demonstrate the effectiveness and superiority of our model.
arXiv commentAccepted by CVPR 2019
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Associative Embedding: End-to-End Learning for Joint Detection and Grouping
- Pixels to Graphs by Associative Embedding
- Visual Relationship Detection with Language Priors
- Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
- Learning to generalize to new compositions in image understanding
- Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
- Referring Relationships
- ViP-CNN: Visual Phrase Guided Convolutional Neural Network