VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Abstract

We propose a Vision-Language Transformer (VLT) framework for referring segmentation to facilitate deep interactions among multi-modal information and enhance the holistic understanding to vision-language features. There are different ways to understand the dynamic emphasis of a language expression, especially when interacting with the image. However, the learned queries in existing transformer works are fixed after training, which cannot cope with the randomness and huge diversity of the language expressions. To address this issue, we propose a Query Generation Module, which dynamically produces multiple sets of input-specific queries to represent the diverse comprehensions of language expression. To find the best among these diverse comprehensions, so as to generate a better mask, we propose a Query Balance Module to selectively fuse the corresponding responses of the set of queries. Furthermore, to enhance the model's ability in dealing with diverse language expressions, we consider inter-sample learning to explicitly endow the model with knowledge of understanding different language expressions to the same object. We introduce masked contrastive learning to narrow down the features of different expressions for the same target object while distinguishing the features of different objects. The proposed approach is lightweight and achieves new state-of-the-art referring segmentation results consistently on five datasets.

arXiv commentTPAMI

References in corpus (13)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Representation Learning with Contrastive Predictive Coding
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Zero-Shot Text-to-Image Generation
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• The 2018 DAVIS Challenge on Video Object Segmentation
• Learning to Compose and Reason with Language Tree Structures for Visual Grounding
• Dynamic Evaluation of Transformer Language Models
• Rethinking Cross-modal Interaction from a Top-down Perspective for Referring Video Object Segmentation
• Comprehensive Multi-Modal Interactions for Referring Image Segmentation
• MaIL: A Unified Mask-Image-Language Trimodal Network for Referring Image Segmentation

Cited by in corpus (5)

• GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models
• Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation
• Efficient Multimodal Large Language Models: A Survey
• AeroReformer: Aerial Referring Transformer for UAV-based Referring Image Segmentation
• Calibration & Reconstruction: Deep Integrated Language for Referring Image Segmentation