A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications

Abstract

Deep learning has revolutionized biomedical research by providing sophisticated methods to handle complex, high-dimensional data. Multimodal deep learning (MDL) further enhances this capability by integrating diverse data types such as imaging, textual data, and genetic information, leading to more robust and accurate predictive models. In MDL, differently from early and late fusion methods, intermediate fusion stands out for its ability to effectively combine modality-specific features during the learning process. This systematic review aims to comprehensively analyze and formalize current intermediate fusion methods in biomedical applications. We investigate the techniques employed, the challenges faced, and potential future directions for advancing intermediate fusion methods. Additionally, we introduce a structured notation to enhance the understanding and application of these methods beyond the biomedical domain. Our findings are intended to support researchers, healthcare professionals, and the broader deep learning community in developing more sophisticated and insightful multimodal models. Through this review, we aim to provide a foundational framework for future research and practical applications in the dynamic field of MDL.

References in corpus (11)

• SMOTE: Synthetic Minority Over-sampling Technique
• A review: Deep learning for medical image segmentation using multi-modality fusion
• Towards Explainable Artificial Intelligence
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Balanced Meta-Softmax for Long-Tailed Visual Recognition
• MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
• A Deep Learning Approach for Virtual Contrast Enhancement in Contrast Enhanced Spectral Mammography
• Multimodal Explainability via Latent Shift applied to COVID-19 stratification
• Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes
• A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1
• Interpretable and Robust AI in EEG Systems: A Survey

Cited by in corpus (13)

• Class Balancing Diversity Multimodal Ensemble for Alzheimer's Disease Diagnosis and Early Detection
• MARIA: a Multimodal Transformer Model for Incomplete Healthcare Data
• Multi-stage intermediate fusion for multimodal learning to classify non-small cell lung cancer subtypes from CT and PET
• Not Another Imputation Method: A Transformer-based Model for Missing Values in Tabular Datasets
• Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging
• XGeM: A Multi-Prompt Foundation Model for Multimodal Medical Data Generation
• Whole-Body Image-to-Image Translation for a Virtual Scanner in a Healthcare Digital Twin
• Timing Is Everything: Finding the Optimal Fusion Points in Multimodal Medical Imaging
• Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
• Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation
• Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages
• Virtual Scanning for NSCLC Histology: Investigating the Discriminatory Power of Synthetic PET
• AudioFuse: Unified Spectral-Temporal Learning via a Hybrid ViT-1D CNN Architecture for Robust Phonocardiogram Classification