FS-Diff: Semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolution
arXiv:2509.09427 · doi:10.1016/j.inffus.2025.103146
Abstract
As an influential information fusion and low-level vision technique, image fusion integrates complementary information from source images to yield an informative fused image. A few attempts have been made in recent years to jointly realize image fusion and super-resolution. However, in real-world applications such as military reconnaissance and long-range detection missions, the target and background structures in multimodal images are easily corrupted, with low resolution and weak semantic information, which leads to suboptimal results in current fusion techniques. In response, we propose FS-Diff, a semantic guidance and clarity-aware joint image fusion and super-resolution method. FS-Diff unifies image fusion and super-resolution as a conditional generation problem. It leverages semantic guidance from the proposed clarity sensing mechanism for adaptive low-resolution perception and cross-modal feature extraction. Specifically, we initialize the desired fused result as pure Gaussian noise and introduce the bidirectional feature Mamba to extract the global features of the multimodal images. Moreover, utilizing the source images and semantics as conditions, we implement a random iterative denoising process via a modified U-Net network. This network istrained for denoising at multiple noise levels to produce high-resolution fusion results with cross-modal features and abundant semantic information. We also construct a powerful aerial view multiscene (AVMS) benchmark covering 600 pairs of images. Extensive joint image fusion and super-resolution experiments on six public and our AVMS datasets demonstrated that FS-Diff outperforms the state-of-the-art methods at multiple magnifications and can recover richer details and semantics in the fused images. The code is available at https://github.com/XylonXu01/FS-Diff.
References in corpus (13)
- Denoising Diffusion Probabilistic Models
- DenseFuse: A Fusion Approach to Infrared and Visible Images
- Diffusion Models in Vision: A Survey
- RFN-Nest: An end-to-end residual fusion network for infrared and visible images
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Blended Diffusion for Text-driven Editing of Natural Images
- MDLatLRR: A novel decomposition method for infrared and visible image fusion
- CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model
- CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion
- Controlling Vision-Language Models for Multi-Task Image Restoration
- Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models
- Saliency Driven Imagery Preprocessing for Efficient Compression -- Industrial Paper