Urban Land Cover Classification with Missing Data Modalities Using Deep Convolutional Neural Networks

Abstract

Automatic urban land cover classification is a fundamental problem in remote sensing, e.g. for environmental monitoring. The problem is highly challenging, as classes generally have high inter-class and low intra-class variance. Techniques to improve urban land cover classification performance in remote sensing include fusion of data from different sensors with different data modalities. However, such techniques require all modalities to be available to the classifier in the decision-making process, i.e. at test time, as well as in training. If a data modality is missing at test time, current state-of-the-art approaches have in general no procedure available for exploiting information from these modalities. This represents a waste of potentially useful information. We propose as a remedy a convolutional neural network (CNN) architecture for urban land cover classification which is able to embed all available training modalities in a so-called hallucination network. The network will in effect replace missing data modalities in the test phase, enabling fusion capabilities even when data modalities are missing in testing. We demonstrate the method using two datasets consisting of optical and digital surface model (DSM) images. We simulate missing modalities by assuming that DSM images are missing during testing. Our method outperforms both standard CNNs trained only on optical images as well as an ensemble of two standard CNNs. We further evaluate the potential of our method to handle situations where only some DSM images are missing during testing. Overall, we show that we can clearly exploit training time information of the missing modality during testing.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Fully Convolutional Networks for Semantic Segmentation
• Dense semantic labeling of sub-decimeter resolution images with convolutional neural networks

Cited by in corpus (10)

• Dense Dilated Convolutions Merging Network for Land Cover Classification
• Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications
• Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks
• SCG-Net: Self-Constructing Graph Neural Networks for Semantic Segmentation
• Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images
• Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
• A Spatially Masked Adaptive Gated Network for multimodal post-flood water extent mapping using SAR and incomplete multispectral data
• On What Depends the Robustness of Multi-source Models to Missing Data in Earth Observation?
• Dense Dilated Convolutions Merging Network for Semantic Mapping of Remote Sensing Images
• Online Sensor Hallucination via Knowledge Distillation for Multimodal Image Classification