Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review

Abstract

Cancer has relational information residing at varying scales, modalities, and resolutions of the acquired data, such as radiology, pathology, genomics, proteomics, and clinical records. Integrating diverse data types can improve the accuracy and reliability of cancer diagnosis and treatment. There can be disease-related information that is too subtle for humans or existing technological tools to discern visually. Traditional methods typically focus on partial or unimodal information about biological systems at individual scales and fail to encapsulate the complete spectrum of the heterogeneous nature of data. Deep neural networks have facilitated the development of sophisticated multimodal data fusion approaches that can extract and integrate relevant information from multiple sources. Recent deep learning frameworks such as Graph Neural Networks (GNNs) and Transformers have shown remarkable success in multimodal learning. This review article provides an in-depth analysis of the state-of-the-art in GNNs and Transformers for multimodal data fusion in oncology settings, highlighting notable research studies and their findings. We also discuss the foundations of multimodal learning, inherent challenges, and opportunities for integrative learning in oncology. By examining the current state and potential future developments of multimodal data integration in oncology, we aim to demonstrate the promising role that multimodal neural networks can play in cancer prevention, early detection, and treatment through informed oncology practices in personalized settings.

References in corpus (38)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Survey on Deep Learning in Medical Image Analysis
• A Comprehensive Survey on Graph Neural Networks
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• On the Opportunities and Risks of Foundation Models
• YFCC100M: The New Data in Multimedia Research
• Modeling polypharmacy side effects with graph convolutional networks
• Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
• Deep Neural Networks and Tabular Data: A Survey
• GNNExplainer: Generating Explanations for Graph Neural Networks
• Attention in Natural Language Processing
• TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification
• Federated Learning Enables Big Data for Rare Cancer Boundary Detection
• Multimodal End-to-End Autonomous Driving
• Transformers in Time-series Analysis: A Tutorial
• Graph Representation Learning in Biomedicine
• Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use case
• ETA Prediction with Graph Neural Networks in Google Maps
• A Survey on Graph-Based Deep Learning for Computational Histopathology
• Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• Graph Neural Networks for Multimodal Single-Cell Data Integration
• Dynamic Graph Representation Learning via Self-Attention Networks
• When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challenges
• RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy
• Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• Zoom in to where it matters: a hierarchical graph based model for mammogram analysis
• Exploring Robust Architectures for Deep Artificial Neural Networks
• Explainability in Graph Neural Networks: An Experimental Survey
• DFTR: Depth-supervised Fusion Transformer for Salient Object Detection
• Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detection
• Vision Through the Veil: Differential Privacy in Federated Learning for Medical Image Classification
• Graph Contrastive Learning for Multi-omics Data
• HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models
• Heterogeneous Graph Neural Networks using Self-supervised Reciprocally Contrastive Learning
• MedAide: Leveraging Large Language Models for On-Premise Medical Assistance on Edge Devices
• Self-Normalizing Foundation Model for Enhanced Multi-Omics Data Analysis in Oncology

Cited by in corpus (3)

• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning
• Performance Analysis and Optimization of 3D Generative Diffusion Models across GPU Architectures