Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection

Abstract

Despite progress in visual perception tasks such as image classification and detection, computers still struggle to understand the interdependency of objects in the scene as a whole, e.g., relations between objects or their attributes. Existing methods often ignore global context cues capturing the interactions among different object instances, and can only recognize a handful of types by exhaustively training individual detectors for all possible relationships. To capture such global interdependency, we propose a deep Variation-structured Reinforcement Learning (VRL) framework to sequentially discover object relationships and attributes in the whole image. First, a directed semantic action graph is built using language priors to provide a rich and compact representation of semantic correlations between object categories, predicates, and attributes. Next, we use a variation-structured traversal over the action graph to construct a small, adaptive action set for each step based on the current state and historical actions. In particular, an ambiguity-aware object mining scheme is used to resolve semantic ambiguity among object categories that the object detector fails to distinguish. We then make sequential predictions using a deep RL framework, incorporating global context cues and semantic embeddings of previously extracted phrases in the state vector. Our experiments on the Visual Relationship Detection (VRD) dataset and the large-scale Visual Genome dataset validate the superiority of VRL, which can achieve significantly better detection results on datasets involving thousands of relationship and attribute types. We also demonstrate that VRL is able to predict unseen types embedded in our action graph by learning correlations on shared graph nodes.

This manuscript is accepted by CVPR 2017 as a spotlight paper

References in corpus (9)

• Playing Atari with Deep Reinforcement Learning
• Deep Residual Learning for Image Recognition
• VQA: Visual Question Answering
• Fully Convolutional Networks for Semantic Segmentation
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
• Visual Relationship Detection with Language Priors
• On Support Relations and Semantic Scene Graphs
• Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates

Cited by in corpus (21)

• Deep Reinforcement Learning: An Overview
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Pixels to Graphs by Associative Embedding
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
• Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning
• Learning to Detect Human-Object Interactions
• Tackling the Challenges in Scene Graph Generation with Local-to-Global Interactions
• Large-Scale Visual Relationship Understanding
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• VrR-VG: Refocusing Visually-Relevant Relationships
• An Interpretable Model for Scene Graph Generation
• LinkNet: Relational Embedding for Scene Graph
• Dynamic-structured Semantic Propagation Network
• Grounded Objects and Interactions for Video Captioning
• Spatial Memory for Context Reasoning in Object Detection
• Adaptive Temporal Encoding Network for Video Instance-level Human Parsing
• Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information
• A2-RL: Aesthetics Aware Reinforcement Learning for Image Cropping
• Target-Tailored Source-Transformation for Scene Graph Generation