Large-Scale Visual Relationship Understanding

Abstract

Large scale visual understanding is challenging, as it requires a model to handle the widely-spread and imbalanced distribution of <subject, relation, object> triples. In real-world scenarios with large numbers of objects and relations, some are seen very commonly while others are barely seen. We develop a new relationship detection model that embeds objects and relations into two vector spaces where both discriminative capability and semantic affinity are preserved. We learn both a visual and a semantic module that map features from the two modalities into a shared space, where matched pairs of features have to discriminate against those unmatched, but also maintain close distances to semantically similar ones. Benefiting from that, our model can achieve superior performance even when the visual entity categories scale up to more than 80,000, with extremely skewed class distribution. We demonstrate the efficacy of our model on a large and imbalanced benchmark based of Visual Genome that comprises 53,000+ objects and 29,000+ relations, a scale at which no previous work has ever been evaluated at. We show superiority of our model over carefully designed baselines on the original Visual Genome dataset with 80,000+ categories. We also show state-of-the-art performance on the VRD dataset and the scene graph dataset which is a subset of Visual Genome with 200 categories.

References in corpus (11)

• Distributed Representations of Words and Phrases and their Compositionality
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Zero-Shot Learning by Convex Combination of Semantic Embeddings
• Pixels to Graphs by Associative Embedding
• Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions
• Order-Embeddings of Images and Language
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
• Information-theoretical label embeddings for large-scale image classification
• Graph R-CNN for Scene Graph Generation
• An Interpretable Model for Scene Graph Generation
• Introduction to the 1st Place Winning Model of OpenImages Relationship Detection Challenge

Cited by in corpus (7)

• VrR-VG: Refocusing Visually-Relevant Relationships
• Visual Semantic Information Pursuit: A Survey
• Scenes and Surroundings: Scene Graph Generation using Relation Transformer
• Leveraging Auxiliary Text for Deep Recognition of Unseen Visual Relationships
• Blocksworld Revisited: Learning and Reasoning to Generate Event-Sequences from Image Pairs
• Target-Tailored Source-Transformation for Scene Graph Generation
• Exploring the Challenges towards Lifelong Fact Learning