Unsupervised and Unregistered Hyperspectral Image Super-Resolution with Mutual Dirichlet-Net
arXiv:1904.12175 · doi:10.1109/TGRS.2021.3079518
Abstract
Hyperspectral images (HSI) provide rich spectral information that contributed to the successful performance improvement of numerous computer vision tasks. However, it can only be achieved at the expense of images' spatial resolution. Hyperspectral image super-resolution (HSI-SR) addresses this problem by fusing low resolution (LR) HSI with multispectral image (MSI) carrying much higher spatial resolution (HR). All existing HSI-SR approaches require the LR HSI and HR MSI to be well registered and the reconstruction accuracy of the HR HSI relies heavily on the registration accuracy of different modalities. This paper exploits the uncharted problem domain of HSI-SR without the requirement of multi-modality registration. Given the unregistered LR HSI and HR MSI with overlapped regions, we design a unique unsupervised learning structure linking the two unregistered modalities by projecting them into the same statistical space through the same encoder. The mutual information (MI) is further adopted to capture the non-linear statistical dependencies between the representations from two modalities (carrying spatial information) and their raw inputs. By maximizing the MI, spatial correlations between different modalities can be well characterized to further reduce the spectral distortion. A collaborative norm is employed as the reconstruction error instead of the more common norm, so that individual pixels can be recovered as accurately as possible. With this design, the network allows to extract correlated spectral and spatial information from unregistered images that better preserves the spectral information. The proposed method is referred to as unregistered and unsupervised mutual Dirichlet Net (-MDN). Extensive experimental results using benchmark HSI datasets demonstrate the superior performance of -MDN as compared to the state-of-the-art.
arXiv commentIEEE Transactions on Remote Sensing and Geoscience
References in corpus (8)
- Point-Set Registration: Coherent Point Drift
- A convex formulation for hyperspectral image superresolution via subspace-based regularization
- Hyperspectral and Multispectral Image Fusion based on a Sparse Representation
- Equitability, mutual information, and the maximal information coefficient
- Fast Fusion of Multi-Band Images Based on Solving a Sylvester Equation
- Coupled Convolutional Neural Network with Adaptive Response Function Learning for Unsupervised Hyperspectral Super-Resolution
- SIRF: Simultaneous Image Registration and Fusion in A Unified Framework
- Recurrent Neural Networks to Correct Satellite Image Classification Maps