RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

Abstract

Pre-trained Vision-Language Models (VLMs) utilizing extensive image-text paired data have demonstrated unprecedented image-text association capabilities, achieving remarkable results across various downstream tasks. A critical challenge is how to make use of existing large-scale pre-trained VLMs, which are trained on common objects, to perform the domain-specific transfer for accomplishing domain-related downstream tasks. A critical challenge is how to make use of existing large-scale pre-trained VLMs, which are trained on common objects, to perform the domain-specific transfer for accomplishing domain-related downstream tasks. In this paper, we propose a new framework that includes the Domain pre-trained Vision-Language Model (DVLM), bridging the gap between the General Vision-Language Model (GVLM) and domain-specific downstream tasks. Moreover, we present an image-text paired dataset in the field of remote sensing (RS), RS5M, which has 5 million RS images with English descriptions. The dataset is obtained from filtering publicly available image-text paired datasets and captioning label-only RS datasets with pre-trained VLM. These constitute the first large-scale RS image-text paired dataset. Additionally, we fine-tuned the CLIP model and tried several Parameter-Efficient Fine-Tuning methods on RS5M to implement the DVLM. Experimental results show that our proposed dataset is highly effective for various tasks, and our model GeoRSCLIP improves upon the baseline or previous state-of-the-art model by $3\%\sim 20\%$ in Zero-shot Classification (ZSC), $3\%\sim 6\%$ in Remote Sensing Cross-Modal Text-Image Retrieval (RSCTIR) and $4\%\sim 5\%$ in Semantic Localization (SeLo) tasks. Dataset and models have been released in: \url{https://github.com/om-ai-lab/RS5M}.

arXiv commentRS5M dataset v5

References in corpus (14)

• Learning to Prompt for Vision-Language Models
• Remote Sensing Image Scene Classification: Benchmark and State of the Art
• AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification
• YFCC100M: The New Data in Multimedia Research
• Exploring Models and Data for Remote Sensing Image Caption Generation
• BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding
• RSVQA: Visual Question Answering for Remote Sensing Data
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• An Empirical Study of Remote Sensing Pretraining
• Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
• Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information
• RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
• Learning to Evaluate Performance of Multi-modal Semantic Localization
• Joint Spatio-Textual Reasoning for Answering Tourism Questions

Cited by in corpus (8)

• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
• Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives
• Mask Approximation Net: A Novel Diffusion Model Approach for Remote Sensing Change Captioning
• RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
• ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAG
• Data-Driven Vision-Language Models for Remote Sensing: A Survey
• Survey on Remote Sensing Scene Classification: From Traditional Methods to Large Generative AI Models