Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval

Abstract

Remote sensing (RS) cross-modal text-image retrieval has attracted extensive attention for its advantages of flexible input and efficient query. However, traditional methods ignore the characteristics of multi-scale and redundant targets in RS image, leading to the degradation of retrieval accuracy. To cope with the problem of multi-scale scarcity and target redundancy in RS multimodal retrieval task, we come up with a novel asymmetric multimodal feature matching network (AMFMN). Our model adapts to multi-scale feature inputs, favors multi-source retrieval methods, and can dynamically filter redundant features. AMFMN employs the multi-scale visual self-attention (MVSA) module to extract the salient features of RS image and utilizes visual features to guide the text representation. Furthermore, to alleviate the positive samples ambiguity caused by the strong intraclass similarity in RS image, we propose a triplet loss function with dynamic variable margin based on prior similarity of sample pairs. Finally, unlike the traditional RS image-text dataset with coarse text and higher intraclass similarity, we construct a fine-grained and more challenging Remote sensing Image-Text Match dataset (RSITMD), which supports RS image retrieval through keywords and sentence separately and jointly. Experiments on four RS text-image datasets demonstrate that the proposed model can achieve state-of-the-art performance in cross-modal RS text-image retrieval task.

References in corpus (8)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Exploring Models and Data for Remote Sensing Image Caption Generation
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval
• On the Arbitrary-Oriented Object Detection: Classification based Approaches Revisited

Cited by in corpus (8)

• Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information
• RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
• RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing
• Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrieval
• Learning to Evaluate Performance of Multi-modal Semantic Localization
• Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives
• Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval
• RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models