MetaSegNet: Metadata-collaborative Vision-Language Representation Learning for Semantic Segmentation of Remote Sensing Images

Abstract

Semantic segmentation of remote sensing images plays a vital role in a wide range of Earth Observation applications, such as land use land cover mapping, environment monitoring, and sustainable development. Driven by rapid developments in artificial intelligence, deep learning (DL) has emerged as the mainstream for semantic segmentation and has achieved many breakthroughs in the field of remote sensing. However, most DL-based methods focus on unimodal visual data while ignoring rich multimodal information involved in the real world. Non-visual data, such as text, can gather extra knowledge from the real world, which can strengthen the interpretability, reliability, and generalization of visual models. Inspired by this, we propose a novel metadata-collaborative segmentation network (MetaSegNet) that applies vision-language representation learning for semantic segmentation of remote sensing images. Unlike the common model structure that only uses unimodal visual data, we extract the key characteristic (e.g. the climate zone) from freely available remote sensing image metadata and transfer it into geographic text prompts via the generic ChatGPT. Then, we construct an image encoder, a text encoder, and a crossmodal attention fusion subnetwork to extract the image and text feature and apply image-text interaction. Benefiting from such a design, the proposed MetaSegNet not only demonstrates superior generalization in zero-shot testing but also achieves competitive accuracy with the state-of-the-art semantic segmentation methods on the large-scale OpenEarthMap dataset (70.4% mIoU) and the Potsdam dataset (93.3% mean F1 score) as well as the LoveDA dataset (52.0% mIoU).

Accepted by IEEE Transactions on Geoscience and Remote Sensing

References in corpus (16)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Deep learning in remote sensing: a review
• ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
• UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery
• ChatGPT: Jack of all trades, master of none
• ABCNet: Attentive Bilateral Contextual Network for Efficient Semantic Segmentation of Fine-Resolution Remote Sensing Images
• A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images
• Improving Semantic Segmentation of Aerial Images Using Patch-based Attention
• Multi-Attention-Network for Semantic Segmentation of Fine Resolution Remote Sensing Images
• A2-FPN for Semantic Segmentation of Fine-Resolution Remotely Sensed Images
• SCAttNet: Semantic Segmentation Network with Spatial and Channel Attention Mechanism for High-Resolution Remote Sensing Images
• LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation
• Dense Dilated Convolutions Merging Network for Land Cover Classification
• Dynamic Multi-Context Segmentation of Remote Sensing Images based on Convolutional Networks
• GeoLLM: Extracting Geospatial Knowledge from Large Language Models
• Large Language Models for Captioning and Retrieving Remote Sensing Images