Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data

Abstract

Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multi-modal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Lidar and hyperspectral dataset provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.

References in corpus (5)

• An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing
• Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification
• CoSpace: Common Subspace Learning from Hyperspectral-Multispectral Correspondences
• Semisupervised Manifold Alignment of Multimodal Remote Sensing Images
• Canonical Correlation Forests

Cited by in corpus (11)

• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
• X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data
• Coupled Convolutional Neural Network with Adaptive Response Function Learning for Unsupervised Hyperspectral Super-Resolution
• Spectral Superresolution of Multispectral Imagery with Joint Sparse and Low-Rank Learning
• Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications
• Nonlocal Patch-Based Fully-Connected Tensor Network Decomposition for Remote Sensing Image Inpainting
• Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model
• Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization
• Joint and Progressive Subspace Analysis (JPSA) with Spatial-Spectral Manifold Alignment for Semi-Supervised Hyperspectral Dimensionality Reduction
• Multi-level Feature Fusion-based CNN for Local Climate Zone Classification from Sentinel-2 Images: Benchmark Results on the So2Sat LCZ42 Dataset
• Spatial-Spectral Manifold Embedding of Hyperspectral Data