Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks

Abstract

Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.

arXiv comment24 pages, 11 figures, submitted to IJRS

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Semi-Supervised Classification with Graph Convolutional Networks
• Fully Convolutional Networks for Semantic Segmentation
• Albumentations: fast and flexible image augmentations
• Squeeze-and-Excitation Networks
• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
• Deep Learning for Classification of Hyperspectral Data: A Comparative Review
• Indoor Semantic Segmentation using depth information
• Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery
• Dense Dilated Convolutions Merging Network for Land Cover Classification
• Efficient piecewise training of deep structured models for semantic segmentation
• Urban Land Cover Classification with Missing Data Modalities Using Deep Convolutional Neural Networks

Cited by in corpus (1)

• Imbalance Knowledge-Driven Multi-modal Network for Land-Cover Semantic Segmentation Using Images and LiDAR Point Clouds