Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training

Abstract

Image-text retrieval is a central problem for understanding the semantic relationship between vision and language, and serves as the basis for various visual and language tasks. Most previous works either simply learn coarse-grained representations of the overall image and text, or elaborately establish the correspondence between image regions or pixels and text words. However, the close relations between coarse- and fine-grained representations for each modality are important for image-text retrieval but almost neglected. As a result, such previous works inevitably suffer from low retrieval accuracy or heavy computational cost. In this work, we address image-text retrieval from a novel perspective by combining coarse- and fine-grained representation learning into a unified framework. This framework is consistent with human cognition, as humans simultaneously pay attention to the entire sample and regional elements to understand the semantic content. To this end, a Token-Guided Dual Transformer (TGDT) architecture which consists of two homogeneous branches for image and text modalities, respectively, is proposed for image-text retrieval. The TGDT incorporates both coarse- and fine-grained retrievals into a unified framework and beneficially leverages the advantages of both retrieval approaches. A novel training objective called Consistent Multimodal Contrastive (CMC) loss is proposed accordingly to ensure the intra- and inter-modal semantic consistencies between images and texts in the common embedding space. Equipped with a two-stage inference method based on the mixed global and local cross-modal similarity, the proposed method achieves state-of-the-art retrieval performances with extremely low inference time when compared with representative recent approaches.

arXiv commentCode is publicly available: https://github.com/LCFractal/TGDT

References in corpus (13)

• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Dual-Path Convolutional Image-Text Embeddings with Instance Loss
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• A Comprehensive Survey of Scene Graphs: Generation and Application
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• Learning Dual Semantic Relations with Graph Attention for Image-Text Matching
• Contrastive Language-Image Pre-Training with Knowledge Graphs