Referring Transformer: A One-step Approach to Multi-task Visual Grounding
arXiv:2106.03089
Abstract
As an important step towards visual reasoning, visual grounding (e.g., phrase localization, referring expression comprehension/segmentation) has been widely explored Previous approaches to referring expression comprehension (REC) or segmentation (RES) either suffer from limited performance, due to a two-stage setup, or require the designing of complex task-specific one-stage architectures. In this paper, we propose a simple one-stage multi-task framework for visual grounding tasks. Specifically, we leverage a transformer architecture, where two modalities are fused in a visual-lingual encoder. In the decoder, the model learns to generate contextualized lingual queries which are then decoded and used to directly regress the bounding box and produce a segmentation mask for the corresponding referred regions. With this simple but highly contextualized model, we outperform state-of-the-arts methods by a large margin on both REC and RES tasks. We also show that a simple pre-training schedule (on an external dataset) further improves the performance. Extensive experiments and ablations illustrate that our model benefits greatly from contextualized information and multi-task training.
References in corpus (7)
- Decoupled Weight Decay Regularization
- YOLOv3: An Incremental Improvement
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- Learning to Compose and Reason with Language Tree Structures for Visual Grounding
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
- Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
- Locate then Segment: A Strong Pipeline for Referring Image Segmentation
Cited by in corpus (5)
- Transformers in Vision: A Survey
- Local-Global Context Aware Transformer for Language-Guided Video Segmentation
- CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding
- Context Disentangling and Prototype Inheriting for Robust Visual Grounding
- HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding