More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

Abstract

Classification and identification of the materials lying over or beneath the Earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS) and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML) -- cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on "what", "where", and "how" to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS datasets. Furthermore, the codes and datasets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning for Hyperspectral Image Classification: An Overview
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing
• Feature Extraction for Hyperspectral Imagery: The Evolution from Shallow to Deep (Overview and Toolbox)
• Cascaded Recurrent Neural Networks for Hyperspectral Image Classification
• Invariant Attribute Profiles: A Spatial-Frequency Joint Feature Extractor for Hyperspectral Image Classification
• Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification
• X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data
• CoSpace: Common Subspace Learning from Hyperspectral-Multispectral Correspondences
• Understanding urban landuse from the above and ground perspectives: a deep learning, multimodal solution
• Fourier-based Rotation-invariant Feature Boosting: An Efficient Framework for Geospatial Object Detection
• Semisupervised Manifold Alignment of Multimodal Remote Sensing Images
• Deep multi-task learning for a geographically-regularized semantic segmentation of aerial images
• MIMA: MAPPER-Induced Manifold Alignment for Semi-Supervised Fusion of Optical Image and Polarimetric SAR Data
• Learning Convolutional Sparse Coding on Complex Domain for Interferometric Phase Restoration
• Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data

Cited by in corpus (50)

• UIU-Net: U-Net in U-Net for Infrared Small Object Detection
• Multimodal Fusion Transformer for Remote Sensing Image Classification
• Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects
• SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery
• Deep Learning for UAV-based Object Detection and Tracking: A Survey
• BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval
• Deep Image Translation with an Affinity-Based Change Prior for Unsupervised Multimodal Change Detection
• Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions
• Nearest Neighbor-Based Contrastive Learning for Hyperspectral and LiDAR Data Classification
• Machine Learning on Neutron and X-Ray Scattering
• Multi-Content Complementation Network for Salient Object Detection in Optical Remote Sensing Images
• FPS-Net: A Convolutional Fusion Network for Large-Scale LiDAR Point Cloud Segmentation
• NFANet: A Novel Method for Weakly Supervised Water Extraction from High-Resolution Remote Sensing Imagery
• Asymmetric Hash Code Learning for Remote Sensing Image Retrieval
• Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection
• Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement
• Imbalance Knowledge-Driven Multi-modal Network for Land-Cover Semantic Segmentation Using Images and LiDAR Point Clouds
• FedFusion: Manifold Driven Federated Learning for Multi-satellite and Multi-modality Fusion
• SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers
• Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications
• Single-stream CNN with Learnable Architecture for Multi-source Remote Sensing Data
• 3D Instance Segmentation of MVS Buildings
• Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks
• Multi-Resolution Data Fusion for Super Resolution Imaging
• Multisource Collaborative Domain Generalization for Cross-Scene Remote Sensing Image Classification
• Convolutional Neural Networks Rarely Learn Shape for Semantic Segmentation
• Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data
• Rotation Equivariant Feature Image Pyramid Network for Object Detection in Optical Remote Sensing Imagery
• SEnSeI: A Deep Learning Module for Creating Sensor Independent Cloud Masks
• Shallow Network Based on Depthwise Over-Parameterized Convolution for Hyperspectral Image Classification
• Triplet-Watershed for Hyperspectral Image Classification
• Automatic Extraction of Road Networks by using Teacher-Student Adaptive Structural Deep Belief Network and Its Application to Landslide Disaster
• On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances and Million-AID
• Learning Cross-Scale Visual Representations for Real-Time Image Geo-Localization
• Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach
• Impact Assessment of Missing Data in Model Predictions for Earth Observation Applications
• Deep learning-based astronomical multimodal data fusion: A comprehensive review
• Bidirectional recurrent imputation and abundance estimation of LULC classes with MODIS multispectral time series and geo-topographic and climatic data
• A Comparative Assessment of Multi-view fusion learning for Crop Classification
• Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model
• 3D object quality prediction for Metal Jet Printer with Multimodal thermal encoder
• A Spatially Masked Adaptive Gated Network for multimodal post-flood water extent mapping using SAR and incomplete multispectral data
• FDCT: Frequency-Aware Decomposition and Cross-Modal Token-Alignment for Multi-Sensor Target Classification
• Joint and Progressive Subspace Analysis (JPSA) with Spatial-Spectral Manifold Alignment for Semi-Supervised Hyperspectral Dimensionality Reduction
• On What Depends the Robustness of Multi-source Models to Missing Data in Earth Observation?
• Multi-patch Feature Pyramid Network for Weakly Supervised Object Detection in Optical Remote Sensing Images
• Evaluating the Role of Training Data Origin for Country-Scale Cropland Mapping in Data-Scarce Regions: A Case Study of Nigeria
• CanadaFireSat: Toward high-resolution wildfire forecasting with multiple modalities
• Multi-modal and frequency-weighted tensor nuclear norm for hyperspectral image denoising
• In the Search for Optimal Multi-view Learning Models for Crop Classification with Global Remote Sensing Data