Visual Relationship Detection with Language Priors

Abstract

Visual relationships capture a wide variety of interactions between pairs of objects in images (e.g. "man riding bicycle" and "man pushing bicycle"). Consequently, the set of possible relationships is extremely large and it is difficult to obtain sufficient training examples for all possible relationships. Because of this limitation, previous work on visual relationship detection has concentrated on predicting only a handful of relationships. Though most relationships are infrequent, their objects (e.g. "man" and "bicycle") and predicates (e.g. "riding" and "pushing") independently occur more frequently. We propose a model that uses this insight to train visual models for objects and predicates individually and later combines them together to predict multiple relationships per image. We improve on prior work by leveraging language priors from semantic word embeddings to finetune the likelihood of a predicted relationship. Our model can scale to predict thousands of types of relationships from a few examples. Additionally, we localize the objects in the predicted relationships as bounding boxes in the image. We further demonstrate that understanding relationships can improve content based image retrieval.

ECCV 2016 Oral

References in corpus (2)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• From Captions to Visual Concepts and Back

Cited by in corpus (54)

• Pixels to Graphs by Associative Embedding
• Detecting Visual Relationships with Deep Relational Networks
• Detecting and Recognizing Human-Object Interactions
• Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
• Exploring Visual Relationship for Image Captioning
• Learning to Compose Dynamic Tree Structures for Visual Contexts
• Scene Graph Generation from Objects, Phrases and Region Captions
• Iterative Visual Reasoning Beyond Convolutions
• Learning to Detect Human-Object Interactions
• Fast Image Caption Generation with Position Alignment
• Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual Attention
• Natural Language Guided Visual Relationship Detection
• Situation Recognition with Graph Neural Networks
• Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
• PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
• Weakly-supervised learning of visual relations
• Scene Graph Generation via Conditional Random Fields
• Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition
• Object Relation Detection Based on One-shot Learning
• Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions
• TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning
• Attentive Relational Networks for Mapping Images to Scene Graphs
• Graph R-CNN for Scene Graph Generation
• A Review on Intelligent Object Perception Methods Combining Knowledge-based Reasoning and Machine Learning
• Dual ResGCN for Balanced Scene GraphGeneration
• Dense Captioning with Joint Inference and Visual Context
• An Interpretable Model for Scene Graph Generation
• Dual-Glance Model for Deciphering Social Relationships
• Scene Graph Parsing as Dependency Parsing
• End-to-End Video Classification with Knowledge Graphs
• LinkNet: Relational Embedding for Scene Graph
• 2nd Place Solution to the GQA Challenge 2019
• The Tensor Brain: Semantic Decoding for Perception and Memory
• Commonly Uncommon: Semantic Sparsity in Situation Recognition
• Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation
• Improving Information Extraction from Images with Learned Semantic Models
• ORD: Object Relationship Discovery for Visual Dialogue Generation
• ViP-CNN: Visual Phrase Guided Convolutional Neural Network
• Explainable Video Action Reasoning via Prior Knowledge and State Transitions
• Recovering the Unbiased Scene Graphs from the Biased Ones
• The Tensor Brain: A Unified Theory of Perception, Memory and Semantic Decoding
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• Generating Holistic 3D Scene Abstractions for Text-based Image Retrieval
• Assisting Scene Graph Generation with Self-Supervision
• Story-oriented Image Selection and Placement
• Seq-SG2SL: Inferring Semantic Layout from Scene Graph Through Sequence to Sequence Learning
• Rb-PaStaNet: A Few-Shot Human-Object Interaction Detection Based on Rules and Part States
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Approximate Query Matching for Image Retrieval
• Dynamic Graph Generation Network: Generating Relational Knowledge from Diagrams
• Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Features
• Better Understanding Hierarchical Visual Relationship for Image Caption
• Addressing Class Imbalance in Scene Graph Parsing by Learning to Contrast and Score