X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data

Abstract

This paper addresses the problem of semi-supervised transfer learning with limited cross-modality data in remote sensing. A large amount of multi-modal earth observation images, such as multispectral imagery (MSI) or synthetic aperture radar (SAR) data, are openly available on a global scale, enabling parsing global urban scenes through remote sensing imagery. However, their ability in identifying materials (pixel-wise classification) remains limited, due to the noisy collection environment and poor discriminative information as well as limited number of well-annotated training images. To this end, we propose a novel cross-modal deep-learning framework, called X-ModalNet, with three well-designed modules: self-adversarial module, interactive learning module, and label propagation module, by learning to transfer more discriminative information from a small-scale hyperspectral image (HSI) into the classification task using a large-scale MSI or SAR data. Significantly, X-ModalNet generalizes well, owing to propagating labels on an updatable graph constructed by high-level features on the top of the network, yielding semi-supervised cross-modality learning. We evaluate X-ModalNet on two multi-modal remote sensing datasets (HSI-MSI and HSI-SAR) and achieve a significant improvement in comparison with several state-of-the-art methods.

References in corpus (28)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• TensorFlow: A system for large-scale machine learning
• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• Fully Convolutional Networks for Semantic Segmentation
• Multi-Scale Context Aggregation by Dilated Convolutions
• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images
• An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing
• Adversarial Feature Learning
• Learning Deconvolution Network for Semantic Segmentation
• Feature Extraction for Hyperspectral Imagery: The Evolution from Shallow to Deep (Overview and Toolbox)
• Cascaded Recurrent Neural Networks for Hyperspectral Image Classification
• Invariant Attribute Profiles: A Spatial-Frequency Joint Feature Extractor for Hyperspectral Image Classification
• Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification
• CoSpace: Common Subspace Learning from Hyperspectral-Multispectral Correspondences
• ORSIm Detector: A Novel Object Detection Framework in Optical Remote Sensing Imagery Using Spatial-Frequency Channel Features
• Understanding urban landuse from the above and ground perspectives: a deep learning, multimodal solution
• Fourier-based Rotation-invariant Feature Boosting: An Efficient Framework for Geospatial Object Detection
• Context Encoding for Semantic Segmentation
• Semisupervised Manifold Alignment of Multimodal Remote Sensing Images
• Multiclass feature learning for hyperspectral image classification: sparse and hierarchical solutions
• Order-Embeddings of Images and Language
• MIMA: MAPPER-Induced Manifold Alignment for Semi-Supervised Fusion of Optical Image and Polarimetric SAR Data
• Learning Convolutional Sparse Coding on Complex Domain for Interferometric Phase Restoration
• Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data
• Change Detection between Multimodal Remote Sensing Data Using Siamese CNN
• FoveaNet: Perspective-aware Urban Scene Parsing

Cited by in corpus (14)

• Graph Convolutional Networks for Hyperspectral Image Classification
• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
• Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects
• Fusion of Dual Spatial Information for Hyperspectral Image Classification
• Multi-Content Complementation Network for Salient Object Detection in Optical Remote Sensing Images
• A Comprehensive Survey for Hyperspectral Image Classification: The Evolution from Conventional to Transformers and Mamba Models
• Optical Remote Sensing Image Understanding with Weak Supervision: Concepts, Methods, and Perspectives
• All Grains, One Scheme (AGOS): Learning Multi-grain Instance Representation for Aerial Scene Classification
• Single-stream CNN with Learnable Architecture for Multi-source Remote Sensing Data
• Exploring Semi-Supervised Learning for Predicting Listener Backchannels
• Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review
• Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data
• Evaluating Self and Semi-Supervised Methods for Remote Sensing Segmentation Tasks
• Joint and Progressive Subspace Analysis (JPSA) with Spatial-Spectral Manifold Alignment for Semi-Supervised Hyperspectral Dimensionality Reduction