Multimodal Deep Learning for Uncertainty-Aware Radiation Pneumonitis Risk Prediction

Abstract

Radiation pneumonitis (RP) is a potentially serious complication of thoracic radiotherapy, making reliable risk prediction important for identifying high-risk patients and guiding effective toxicity management. Although deep learning models can predict RP risk, uncertainty quantification remains underdeveloped in radiation toxicity assessment. We developed a Multimodal Bayesian Diffusion Transformer (MM-DiT) that integrates pretreatment computed tomography (CT) and three-dimensional planning dose distributions for RP risk prediction and uncertainty quantification. MM-DiT combines self-supervised multimodal representation learning with Bayesian classification to estimate individual risk and characterize aleatoric, epistemic, and label uncertainty. In independent Mount Sinai Health System (MSHS) and RTOG cohorts, MM-DiT achieved areas under the receiver operating characteristic curve (AUCs) of $0.89$ and $0.86$, respectively, with expected calibration errors of $0.02$ and $0.03$. Incorporating post-treatment CT increased AUC to $0.95$ in MSHS and reduced aleatoric and epistemic uncertainty by approximately $40-45\%$ and $40\%$, respectively. These findings establish a framework for uncertainty-aware radiation toxicity prediction that complements risk estimates with information about predictive reliability. By integrating multimodal learning and uncertainty quantification, MM-DiT may advance more interpretable and trustworthy AI-assisted toxicity assessment and support individualized radiotherapy planning.

arXiv comment28 pages, 9 figures, 5 tables