ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss via Meta-Learning

Abstract

Image fusion aims to combine information from multiple source images into a single one with more comprehensive informational content. Deep learning-based image fusion algorithms face significant challenges, including the lack of a definitive ground truth and the corresponding distance measurement. Additionally, current manually defined loss functions limit the model's flexibility and generalizability for various fusion tasks. To address these limitations, we propose ReFusion, a unified meta-learning based image fusion framework that dynamically optimizes the fusion loss for various tasks through source image reconstruction. Compared to existing methods, ReFusion employs a parameterized loss function, that allows the training framework to be dynamically adapted according to the specific fusion scenario and task. ReFusion consists of three key components: a fusion module, a source reconstruction module, and a loss proposal module. We employ a meta-learning strategy to train the loss proposal module using the reconstruction loss. This strategy forces the fused image to be more conducive to reconstruct source images, allowing the loss proposal module to generate a adaptive fusion loss that preserves the optimal information from the source images. The update of the fusion module relies on the learnable fusion loss proposed by the loss proposal module. The three modules update alternately, enhancing each other to optimize the fusion loss for different tasks and consistently achieve satisfactory results. Extensive experiments demonstrate that ReFusion is capable of adapting to various tasks, including infrared-visible, medical, multi-focus, and multi-exposure image fusion.

This article is published in International Journal of Computer Vision (IJCV), 2024

References in corpus (9)

• A Survey on Deep Learning in Medical Image Analysis
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• DenseFuse: A Fusion Approach to Infrared and Visible Images
• RFN-Nest: An end-to-end residual fusion network for infrared and visible images
• Medical Image Fusion: A survey of the state of the art
• Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting
• DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion
• CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion
• Real-MFF: A Large Realistic Multi-focus Image Dataset with Ground Truth