Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval
arXiv:2411.14704 · doi:10.1109/TGRS.2024.3489224
Abstract
Remote sensing cross-modal text-image retrieval (RSCTIR) has gained attention for its utility in information mining. However, challenges remain in effectively integrating global and local information due to variations in remote sensing imagery and ensuring proper feature pre-alignment before modal fusion, which affects retrieval accuracy and efficiency. To address these issues, we propose CMPAGL, a cross-modal pre-aligned method leveraging global and local information. Our Gswin transformer block combines local window self-attention and global-local window cross-attention to capture multi-scale features. A pre-alignment mechanism simplifies modal fusion training, improving retrieval performance. Additionally, we introduce a similarity matrix reweighting (SMR) algorithm for reranking, and enhance the triplet loss function with an intra-class distance term to optimize feature learning. Experiments on four datasets, including RSICD and RSITMD, validate CMPAGL's effectiveness, achieving up to 4.65% improvement in R@1 and 2.28% in mean Recall (mR) over state-of-the-art methods.
References in corpus (10)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Exploring Models and Data for Remote Sensing Image Caption Generation
- Dual-Path Convolutional Image-Text Embeddings with Instance Loss
- VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
- Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information
- Understanding urban landuse from the above and ground perspectives: a deep learning, multimodal solution
- Multitask Learning for SAR Ship Detection with Gaussian-Mask Joint Segmentation
- Cross Spectral Image Reconstruction Using a Deep Guided Neural Network