Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts

Abstract

Textual grounding is an important but challenging task for human-computer interaction, robotics and knowledge mining. Existing algorithms generally formulate the task as selection from a set of bounding box proposals obtained from deep net based systems. In this work, we demonstrate that we can cast the problem of textual grounding into a unified framework that permits efficient search over all possible bounding boxes. Hence, the method is able to consider significantly more proposals and doesn't rely on a successful first stage hypothesizing bounding box proposals. Beyond, we demonstrate that the trained parameters of our model can be used as word-embeddings which capture spatial-image relationships and provide interpretability. Lastly, at the time of submission, our approach outperformed the current state-of-the-art methods on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77% respectively.

Accepted to NIPS 2017

References in corpus (4)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• A Joint Model of Language and Perception for Grounded Attribute Learning
• Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

Cited by in corpus (2)

• A Fast and Accurate One-Stage Approach to Visual Grounding
• Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding