Visual Translation Embedding Network for Visual Relation Detection

Abstract

Visual relations, such as "person ride bike" and "bike next to car", offer a comprehensive scene understanding of an image, and have already shown their great utility in connecting computer vision and natural language. However, due to the challenging combinatorial complexity of modeling subject-predicate-object relation triplets, very little work has been done to localize and predict visual relations. Inspired by the recent advances in relational representation learning of knowledge bases and convolutional object detection networks, we propose a Visual Translation Embedding network (VTransE) for visual relation detection. VTransE places objects in a low-dimensional relation space where a relation can be modeled as a simple vector translation, i.e., subject + predicate $\approx$ object. We propose a novel feature extraction layer that enables object-relation knowledge transfer in a fully-convolutional fashion that supports training and inference in a single forward/backward pass. To the best of our knowledge, VTransE is the first end-to-end relation detection network. We demonstrate the effectiveness of VTransE over other state-of-the-art methods on two large-scale datasets: Visual Relationship and Visual Genome. Note that even though VTransE is a purely visual model, it is still competitive to the Lu's multi-modal model with language priors.

References in corpus (5)

• Deep Residual Learning for Image Recognition
• VQA: Visual Question Answering
• DRAW: A Recurrent Neural Network For Image Generation
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Learning to generalize to new compositions in image understanding

Cited by in corpus (26)

• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Pixels to Graphs by Associative Embedding
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi- Supervised Semantic Segmentation
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Scene Graph Generation from Objects, Phrases and Region Captions
• Learning to Detect Human-Object Interactions
• Auto-Encoding Scene Graphs for Image Captioning
• Natural Language Guided Visual Relationship Detection
• Situation Recognition with Graph Neural Networks
• PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
• Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks
• Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
• Object Relation Detection Based on One-shot Learning
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Crowd Counting Using Scale-Aware Attention Networks
• Learning Cross-modal Context Graph for Visual Grounding
• An Interpretable Model for Scene Graph Generation
• Dual ResGCN for Balanced Scene GraphGeneration
• 2nd Place Solution to the GQA Challenge 2019
• Grounded Objects and Interactions for Video Captioning
• Large-Scale Attribute-Object Compositions
• Improving Information Extraction from Images with Learned Semantic Models
• simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions
• Topic-Guided Attention for Image Captioning
• Target-Tailored Source-Transformation for Scene Graph Generation