RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
arXiv:2408.14744 · doi:10.1016/j.isprsjprs.2025.05.002
Abstract
Abundant, well-annotated multimodal data in remote sensing are pivotal for aligning complex visual remote sensing (RS) scenes with human language, enabling the development of specialized vision language models across diverse RS interpretation tasks. However, annotating RS images with rich linguistic semantics at scale demands expertise in RS and substantial human labor, making it costly and often impractical. In this study, we propose a workflow that leverages large language models (LLMs) to generate multimodal datasets with semantically rich captions at scale from plain OpenStreetMap (OSM) data for images sourced from the Google Earth Engine (GEE) platform. This approach facilitates the generation of paired remote sensing data and can be readily scaled up using openly available data. Within this framework, we present RSTeller, a multimodal dataset comprising over 1.3 million RS images, each accompanied by two descriptive captions. Extensive experiments demonstrate that RSTeller enhances the performance of multiple existing vision language models for RS scene understanding through continual pre-training. Our methodology significantly reduces the manual effort and expertise needed for annotating remote sensing imagery while democratizing access to high-quality annotated data. This advancement fosters progress in visual language modeling and encourages broader participation in remote sensing research and applications. The RSTeller dataset is available at https://github.com/SlytherinGe/RSTeller.
Published on ISPRS, minor typos corrected
References in corpus (24)
- Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- LLaMA: Open and Efficient Foundation Language Models
- Remote Sensing Image Scene Classification: Benchmark and State of the Art
- AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification
- YFCC100M: The New Data in Multimedia Research
- Scaling Laws for Neural Language Models
- Visual Instruction Tuning
- Exploring Models and Data for Remote Sensing Image Caption Generation
- Reproducible scaling laws for contrastive language-image learning
- PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrieval
- CoCa: Contrastive Captioners are Image-Text Foundation Models
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
- Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
- PaLI: A Jointly-Scaled Multilingual Language-Image Model
- RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data
- RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing
- CogVLM: Visual Expert for Pretrained Language Models
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
- RSGPT: A Remote Sensing Vision Language Model and Benchmark
- Improving CLIP Training with Language Rewrites
- Caption Anything: Interactive Image Description with Diverse Multimodal Controls