Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer

Abstract

This study proposes a novel approach combining Multimodal Deep Learning with intrinsic eXplainable Artificial Intelligence techniques to predict pathological response in non-small cell lung cancer patients undergoing neoadjuvant therapy. Due to the limitations of existing radiomics and unimodal deep learning approaches, we introduce an intermediate fusion strategy that integrates imaging and clinical data, enabling efficient interaction between data modalities. The proposed Multimodal Doctor-in-the-Loop method further enhances clinical relevance by embedding clinicians' domain knowledge directly into the training process, guiding the model's focus gradually from broader lung regions to specific lesions. Results demonstrate improved predictive accuracy and explainability, providing insights into optimal data integration strategies for clinical applications.

arXiv admin note: substantial text overlap with arXiv:2502.17503

References in corpus (7)

• A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications
• Multimodal Explainability via Latent Shift applied to COVID-19 stratification
• Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes
• Class Balancing Diversity Multimodal Ensemble for Alzheimer's Disease Diagnosis and Early Detection
• A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1
• Doctor-in-the-Loop: An Explainable, Multi-View Deep Learning Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
• Multi-Dataset Multi-Task Learning for COVID-19 Prognosis