Changes to Captions: An Attentive Network for Remote Sensing Change Captioning

Abstract

In recent years, advanced research has focused on the direct learning and analysis of remote sensing images using natural language processing (NLP) techniques. The ability to accurately describe changes occurring in multi-temporal remote sensing images is becoming increasingly important for geospatial understanding and land planning. Unlike natural image change captioning tasks, remote sensing change captioning aims to capture the most significant changes, irrespective of various influential factors such as illumination, seasonal effects, and complex land covers. In this study, we highlight the significance of accurately describing changes in remote sensing images and present a comparison of the change captioning task for natural and synthetic images and remote sensing images. To address the challenge of generating accurate captions, we propose an attentive changes-to-captions network, called Chg2Cap for short, for bi-temporal remote sensing images. The network comprises three main components: 1) a Siamese CNN-based feature extractor to collect high-level representations for each image pair; 2) an attentive decoder that includes a hierarchical self-attention block to locate change-related features and a residual block to generate the image embedding; and 3) a transformer-based caption generator to decode the relationship between the image embedding and the word embedding into a description. The proposed Chg2Cap network is evaluated on two representative remote sensing datasets, and a comprehensive experimental analysis is provided. The code and pre-trained models will be available online at https://github.com/ShizhenChang/Chg2Cap.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Exploring Models and Data for Remote Sensing Image Caption Generation
• Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion
• Consistency-Regularized Region-Growing Network for Semantic Segmentation of Urban Scenes with Point-Level Annotations
• Txt2Img-MHN: Remote Sensing Image Generation from Text Using Modern Hopfield Networks
• HAR-Net: Joint Learning of Hybrid Attention for Single-stage Object Detection
• CLIP4IDC: CLIP for Image Difference Captioning

Cited by in corpus (8)

• Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis
• Txt2Img-MHN: Remote Sensing Image Generation from Text Using Modern Hopfield Networks
• A Lightweight Sparse Focus Transformer for Remote Sensing Image Change Captioning
• Remote Sensing SpatioTemporal Vision-Language Models: A Comprehensive Survey
• Mask Approximation Net: A Novel Diffusion Model Approach for Remote Sensing Change Captioning
• RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering
• Robust Change Captioning in Remote Sensing: SECOND-CC Dataset and MModalCC Framework
• Forest-Chat: Adapting Vision-Language Agents for Interactive Forest Change Analysis