Natural Language Guided Visual Relationship Detection
arXiv:1711.06032
Abstract
Reasoning about the relationships between object pairs in images is a crucial task for holistic scene understanding. Most of the existing works treat this task as a pure visual classification task: each type of relationship or phrase is classified as a relation category based on the extracted visual features. However, each kind of relationships has a wide variety of object combination and each pair of objects has diverse interactions. Obtaining sufficient training samples for all possible relationship categories is difficult and expensive. In this work, we propose a natural language guided framework to tackle this problem. We propose to use a generic bi-directional recurrent neural network to predict the semantic connection between the participating objects in the relationship from the aspect of natural language. The proposed simple method achieves the state-of-the-art on the Visual Relationship Detection (VRD) and Visual Genome datasets, especially when predicting unseen relationships (e.g. recall improved from 76.42% to 89.79% on VRD zero-shot testing set).
added supplementary material
References in corpus (13)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- VQA: Visual Question Answering
- Fully Convolutional Networks for Semantic Segmentation
- Pyramid Scene Parsing Network
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Visual Relationship Detection with Language Priors
- Scene Graph Generation by Iterative Message Passing
- Detecting Visual Relationships with Deep Relational Networks
- Visual Translation Embedding Network for Visual Relation Detection
- Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
- Attribute-Graph: A Graph based approach to Image Ranking