Multimodal Deep Unfolding for Guided Image Super-Resolution

Abstract

The reconstruction of a high resolution image given a low resolution observation is an ill-posed inverse problem in imaging. Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a high-resolution output. Unlike existing deep multimodal models that do not incorporate domain knowledge about the problem, we propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information; therefore, the proposed neural network is interpretable by design. The deep unfolding architecture is used as a core component of a multimodal framework for guided image super-resolution. An alternative multimodal design is investigated by employing residual learning to improve the training efficiency. The presented multimodal approach is applied to super-resolution of near-infrared and multi-spectral images as well as depth upsampling using RGB images as side information. Experimental results show that our model outperforms state-of-the-art methods.

References in corpus (7)

• Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections
• AMP-Inspired Deep Networks for Sparse Linear Inverse Problems
• Learned Convolutional Sparse Coding
• Multimodal Image Super-resolution via Joint Sparse Representations induced by Coupled Dictionaries
• Multi-modal dictionary learning for image separation with application in art investigation
• Deep Coupled-Representation Learning for Sparse Linear Inverse Problems with Side Information
• X-ray image separation via coupled dictionary learning

Cited by in corpus (1)

• Interpretable Deep Multimodal Image Super-Resolution