VrR-VG: Refocusing Visually-Relevant Relationships

Abstract

Relationships encode the interactions among individual instances, and play a critical role in deep visual scene understanding. Suffering from the high predictability with non-visual information, existing methods tend to fit the statistical bias rather than ``learning'' to ``infer'' the relationships from images. To encourage further development in visual relationships, we propose a novel method to automatically mine more valuable relationships by pruning visually-irrelevant ones. We construct a new scene-graph dataset named Visually-Relevant Relationships Dataset (VrR-VG) based on Visual Genome. Compared with existing datasets, the performance gap between learnable and statistical method is more significant in VrR-VG, and frequency-based analysis does not work anymore. Moreover, we propose to learn a relationship-aware representation by jointly considering instances, attributes and relationships. By applying the representation-aware feature learned on VrR-VG, the performances of image captioning and visual question answering are systematically improved with a large margin, which demonstrates the gain of our dataset and the features embedding schema. VrR-VG is available via http://vrr-vg.com/.

Accepted by ICCV2019

References in corpus (11)

• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Aggregated Residual Transformations for Deep Neural Networks
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Pixels to Graphs by Associative Embedding
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
• Boosting Image Captioning with Attributes
• Large-Scale Visual Relationship Understanding

Cited by in corpus (5)

• Privacy Intelligence: A Survey on Image Privacy in Online Social Networks
• An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
• Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
• Character Matters: Video Story Understanding with Character-Aware Relations
• HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos