Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation

Abstract

Understanding visual relationships involves identifying the subject, the object, and a predicate relating them. We leverage the strong correlations between the predicate and the (subj,obj) pair (both semantically and spatially) to predict the predicates conditioned on the subjects and the objects. Modeling the three entities jointly more accurately reflects their relationships, but complicates learning since the semantic space of visual relationships is huge and the training data is limited, especially for the long-tail relationships that have few instances. To overcome this, we use knowledge of linguistic statistics to regularize visual model learning. We obtain linguistic knowledge by mining from both training annotations (internal knowledge) and publicly available text, e.g., Wikipedia (external knowledge), computing the conditional probability distribution of a predicate given a (subj,obj) pair. Then, we distill the knowledge into a deep model to achieve better generalization. Our experimental results on the Visual Relationship Detection (VRD) and Visual Genome datasets suggest that with this linguistic knowledge distillation, our model outperforms the state-of-the-art methods significantly, especially when predicting unseen relationships (e.g., recall improved from 8.45% to 19.17% on VRD zero-shot testing set).

ICCV 2017

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Composing graphical models with neural networks for structured representations and fast inference
• When Are Tree Structures Necessary for Deep Learning of Representations?
• Visual Relationship Detection with Language Priors
• Detecting Visual Relationships with Deep Relational Networks
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
• The Role of Context Selection in Object Detection
• Generating Holistic 3D Scene Abstractions for Text-based Image Retrieval

Cited by in corpus (13)

• NISP: Pruning Networks using Neuron Importance Score Propagation
• Neural Compatibility Modeling with Attentive Knowledge Distillation
• Natural Language Guided Visual Relationship Detection
• Dynamic Zoom-in Network for Fast Object Detection in Large Images
• Modeling Local Geometric Structure of 3D Point Clouds using Geo-CNN
• Character Matters: Video Story Understanding with Character-Aware Relations
• An Interpretable Model for Scene Graph Generation
• Deep Regionlets for Object Detection
• 2nd Place Solution to the GQA Challenge 2019
• LinkNet: Relational Embedding for Scene Graph
• Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search
• Learning Predicates as Functions to Enable Few-shot Scene Graph Prediction
• Give Me Something to Eat: Referring Expression Comprehension with Commonsense Knowledge