RFN-Nest: An end-to-end residual fusion network for infrared and visible images

Abstract

In the image fusion field, the design of deep learning-based fusion methods is far from routine. It is invariably fusion-task specific and requires a careful consideration. The most difficult part of the design is to choose an appropriate strategy to generate the fused image for a specific task in hand. Thus, devising learnable fusion strategy is a very challenging problem in the community of image fusion. To address this problem, a novel end-to-end fusion network architecture (RFN-Nest) is developed for infrared and visible image fusion. We propose a residual fusion network (RFN) which is based on a residual architecture to replace the traditional fusion approach. A novel detail-preserving loss function, and a feature enhancing loss function are proposed to train RFN. The fusion model learning is accomplished by a novel two-stage training strategy. In the first stage, we train an auto-encoder based on an innovative nest connection (Nest) concept. Next, the RFN is trained using the proposed loss functions. The experimental results on public domain data sets show that, compared with the existing methods, our end-to-end fusion network delivers a better performance than the state-of-the-art methods in both subjective and objective evaluation. The code of our fusion method is available at https://github.com/hli1221/imagefusion-rfn-nest

Accepted by Information Fusion. 17 pages, 18 figures, 8 tables

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models
• AFAT: Adaptive Failure-Aware Tracker for Robust Visual Object Tracking

Cited by in corpus (13)

• CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach
• CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion
• ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss via Meta-Learning
• Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection
• FS-Diff: Semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolution
• Breaking Free from Fusion Rule: A Fully Semantic-driven Infrared and Visible Image Fusion
• An Attention-Guided and Wavelet-Constrained Generative Adversarial Network for Infrared and Visible Image Fusion
• A Unified Multi-Task Learning Framework of Real-Time Drone Supervision for Crowd Counting
• Deep Unfolding Multi-modal Image Fusion Network via Attribution Analysis
• MMDRFuse: Distilled Mini-Model with Dynamic Refresh for Multi-Modality Image Fusion
• Visible and Infrared Image Fusion Using Encoder-Decoder Network
• Dynamic Brightness Adaptation for Robust Multi-modal Image Fusion
• Dynamic Image Restoration and Fusion Based on Dynamic Degradation