A Fast and Accurate One-Stage Approach to Visual Grounding

Abstract

We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight. The performances of existing propose-and-rank two-stage methods are capped by the quality of the region candidates they propose in the first stage --- if none of the candidates could cover the ground truth region, there is no hope in the second stage to rank the right region to the top. To avoid this caveat, we propose a one-stage model that enables end-to-end joint optimization. The main idea is as straightforward as fusing a text query's embedding into the YOLOv3 object detector, augmented by spatial features so as to account for spatial mentions in the query. Despite being simple, this one-stage approach shows great potential in terms of both accuracy and speed for both phrase localization and referring expression comprehension, according to our experiments. Given these results along with careful investigations into some popular region proposals, we advocate for visual grounding a paradigm shift from the conventional two-stage methods to the one-stage framework.

arXiv commentICCV 2019 Oral

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• YOLOv3: An Incremental Improvement
• Deep Residual Learning for Image Recognition
• YOLO9000: Better, Faster, Stronger
• Image Captioning with Semantic Attention
• Generation and Comprehension of Unambiguous Object Descriptions
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• MAttNet: Modular Attention Network for Referring Expression Comprehension
• Learning Deep Structure-Preserving Image-Text Embeddings
• Modeling Context in Referring Expressions
• Visual7W: Grounded Question Answering in Images
• Segmentation from Natural Language Expressions
• Natural Language Object Retrieval
• Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
• Unsupervised Image Captioning
• A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
• Dynamic Multimodal Instance Segmentation guided by natural language queries