Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review

Abstract

Cancer has relational information residing at varying scales, modalities, and resolutions of the acquired data, such as radiology, pathology, genomics, proteomics, and clinical records. Integrating diverse data types can improve the accuracy and reliability of cancer diagnosis and treatment. There can be disease-related information that is too subtle for humans or existing technological tools to discern visually. Traditional methods typically focus on partial or unimodal information about biological systems at individual scales and fail to encapsulate the complete spectrum of the heterogeneous nature of data. Deep neural networks have facilitated the development of sophisticated multimodal data fusion approaches that can extract and integrate relevant information from multiple sources. Recent deep learning frameworks such as Graph Neural Networks (GNNs) and Transformers have shown remarkable success in multimodal learning. This review article provides an in-depth analysis of the state-of-the-art in GNNs and Transformers for multimodal data fusion in oncology settings, highlighting notable research studies and their findings. We also discuss the foundations of multimodal learning, inherent challenges, and opportunities for integrative learning in oncology. By examining the current state and potential future developments of multimodal data integration in oncology, we aim to demonstrate the promising role that multimodal neural networks can play in cancer prevention, early detection, and treatment through informed oncology practices in personalized settings.

References in corpus (21)

• Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
• Deep Neural Networks and Tabular Data: A Survey
• TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification
• Federated Learning Enables Big Data for Rare Cancer Boundary Detection
• Transformers in Time-series Analysis: A Tutorial
• ETA Prediction with Graph Neural Networks in Google Maps
• Graph Neural Networks for Multimodal Single-Cell Data Integration
• When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challenges
• RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy
• Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• Zoom in to where it matters: a hierarchical graph based model for mammogram analysis
• Explainability in Graph Neural Networks: An Experimental Survey
• DFTR: Depth-supervised Fusion Transformer for Salient Object Detection
• Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detection
• Vision Through the Veil: Differential Privacy in Federated Learning for Medical Image Classification
• Graph Contrastive Learning for Multi-omics Data
• Heterogeneous Graph Neural Networks using Self-supervised Reciprocally Contrastive Learning
• HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models
• MedAide: Leveraging Large Language Models for On-Premise Medical Assistance on Edge Devices
• Self-Normalizing Foundation Model for Enhanced Multi-Omics Data Analysis in Oncology

Cited by in corpus (3)

• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning
• Performance Analysis and Optimization of 3D Generative Diffusion Models across GPU Architectures