Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications

Abstract

The advances in remote sensing technologies have boosted applications for Earth observation. These technologies provide multiple observations or views with different levels of information. They might contain static or temporary views with different levels of resolution, in addition to having different types and amounts of noise due to sensor calibration or deterioration. A great variety of deep learning models have been applied to fuse the information from these multiple views, known as deep multi-view or multi-modal fusion learning. However, the approaches in the literature vary greatly since different terminology is used to refer to similar concepts or different illustrations are given to similar techniques. This article gathers works on multi-view fusion for Earth observation by focusing on the common practices and approaches used in the literature. We summarize and structure insights from several different publications concentrating on unifying points and ideas. In this manuscript, we provide a harmonized terminology while at the same time mentioning the various alternative terms that are used in literature. The topics covered by the works reviewed focus on supervised learning with the use of neural network models. We hope this review, with a long list of recent references, can support future research and lead to a unified advance in the area.

appendix with additional tables. Preprint submitted to journal

References in corpus (13)

• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
• Using Transfer Learning for Image-Based Cassava Disease Detection
• A CNN-RNN Framework for Crop Yield Prediction
• Classification of Hyperspectral and LiDAR Data Using Coupled CNNs
• Machine Learning Information Fusion in Earth Observation: A Comprehensive Review of Methods, Applications and Data Sources
• BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval
• Understanding urban landuse from the above and ground perspectives: a deep learning, multimodal solution
• SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal
• Deep Multimodal Fusion by Channel Exchanging
• Adjacent-Level Feature Cross-Fusion With 3-D CNN for Remote Sensing Image Change Detection
• Revisiting Consistency Regularization for Semi-supervised Change Detection in Remote Sensing Images
• MultiEarth 2023 -- Multimodal Learning for Earth and Environment Workshop and Challenge
• Corn Yield Prediction with Ensemble CNN-DNN

Cited by in corpus (9)

• Adaptive Fusion of Multi-view Remote Sensing data for Optimal Sub-field Crop Yield Prediction
• Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach
• Impact Assessment of Missing Data in Model Predictions for Earth Observation Applications
• A Comparative Assessment of Multi-view fusion learning for Crop Classification
• Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
• A Spatially Masked Adaptive Gated Network for multimodal post-flood water extent mapping using SAR and incomplete multispectral data
• On What Depends the Robustness of Multi-source Models to Missing Data in Earth Observation?
• In the Search for Optimal Multi-view Learning Models for Crop Classification with Global Remote Sensing Data
• An Analysis of Temporal Dropout in Earth Observation Time Series for Regression Tasks