An End-to-End Approach to Natural Language Object Retrieval via Context-Aware Deep Reinforcement Learning

Abstract

We propose an end-to-end approach to the natural language object retrieval task, which localizes an object within an image according to a natural language description, i.e., referring expression. Previous works divide this problem into two independent stages: first, compute region proposals from the image without the exploration of the language description; second, score the object proposals with regard to the referring expression and choose the top-ranked proposals. The object proposals are generated independently from the referring expression, which makes the proposal generation redundant and even irrelevant to the referred object. In this work, we train an agent with deep reinforcement learning, which learns to move and reshape a bounding box to localize the object according to the referring expression. We incorporate both the spatial and temporal context information into the training procedure. By simultaneously exploiting local visual information, the spatial and temporal context and the referring language a priori, the agent selects an appropriate action to take at each time. A special action is defined to indicate when the agent finds the referred object, and terminate the procedure. We evaluate our model on various datasets, and our algorithm significantly outperforms the compared algorithms. Notably, the accuracy improvement of our method over the recent method GroundeR and SCRC on the ReferItGame dataset are 7.67% and 18.25%, respectively.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: A system for large-scale machine learning
• Deep Residual Learning for Image Recognition
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Going Deeper with Convolutions
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Show and Tell: A Neural Image Caption Generator
• Generation and Comprehension of Unambiguous Object Descriptions
• Modeling Context in Referring Expressions
• Natural Language Object Retrieval

Cited by in corpus (3)

• A Better Loss for Visual-Textual Grounding
• Semi Supervised Phrase Localization in a Bidirectional Caption-Image Retrieval Framework
• Natural Language Person Search Using Deep Reinforcement Learning