M${}^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data

Abstract

Multimodal electronic health record (EHR) data are widely used in clinical applications. Conventional methods usually assume that each sample (patient) is associated with the unified observed modalities, and all modalities are available for each sample. However, missing modality caused by various clinical and social reasons is a common issue in real-world clinical scenarios. Existing methods mostly rely on solving a generative model that learns a mapping from the latent space to the original input space, which is an unstable ill-posed inverse problem. To relieve the underdetermined system, we propose a model solving a direct problem, dubbed learning with Missing Modalities in Multimodal healthcare data (M3Care). M3Care is an end-to-end model compensating the missing information of the patients with missing modalities to perform clinical analysis. Instead of generating raw missing data, M3Care imputes the task-related information of the missing modalities in the latent space by the auxiliary information from each patient's similar neighbors, measured by a task-guided modality-adaptive similarity metric, and thence conducts the clinical tasks. The task-guided modality-adaptive similarity metric utilizes the uncensored modalities of the patient and the other patients who also have the same uncensored modalities to find similar patients. Experiments on real-world datasets show that M3Care outperforms the state-of-the-art baselines. Moreover, the findings discovered by M3Care are consistent with experts and medical knowledge, demonstrating the capability and the potential of providing useful insights and explanations.

arXiv comment9 pages

References in corpus (7)

• A Survey on Multi-view Learning
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
• Learning Deep Kernels for Non-Parametric Two-Sample Tests
• RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data
• Top-N Recommendation on Graphs

Cited by in corpus (8)

• DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency
• Formalizing Multimedia Recommendation through Multimodal Deep Learning
• ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaboration
• EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation
• MARIA: a Multimodal Transformer Model for Incomplete Healthcare Data
• FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Prediction
• Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning
• PRISM: Mitigating EHR Data Sparsity via Learning from Missing Feature Calibrated Prototype Patient Representations