From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine

Abstract

Generative artificial intelligence (AI) models, such as diffusion models and OpenAI's ChatGPT, are transforming medicine by enhancing diagnostic accuracy and automating clinical workflows. The field has advanced rapidly, evolving from text-only large language models for tasks such as clinical documentation and decision support to multimodal AI systems capable of integrating diverse data modalities, including imaging, text, and structured data, within a single model. The diverse landscape of these technologies, along with rising interest, highlights the need for a comprehensive review of their applications and potential. This scoping review explores the evolution of multimodal AI, highlighting its methods, applications, datasets, and evaluation in clinical settings. Adhering to PRISMA-ScR guidelines, we systematically queried PubMed, IEEE Xplore, and Web of Science, prioritizing recent studies published up to the end of 2024. After rigorous screening, 144 papers were included, revealing key trends and challenges in this dynamic field. Our findings underscore a shift from unimodal to multimodal approaches, driving innovations in diagnostic support, medical report generation, drug discovery, and conversational AI. However, critical challenges remain, including the integration of heterogeneous data types, improving model interpretability, addressing ethical concerns, and validating AI systems in real-world clinical settings. This review summarizes the current state of the art, identifies critical gaps, and provides insights to guide the development of scalable, trustworthy, and clinically impactful multimodal AI solutions in healthcare.

References in corpus (32)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Learning Transferable Visual Models From Natural Language Supervision
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization
• A Survey on Multimodal Large Language Models
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
• Interactive and Explainable Region-guided Radiology Report Generation
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
• Large Language Models Streamline Automated Machine Learning for Clinical Studies
• MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data
• Foundation Models in Radiology: What, How, When, Why and Why Not
• Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning
• Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
• GREEN: Generative Radiology Report Evaluation and Error Notation
• Best Practices for Large Language Models in Radiology
• Advancing Multimodal Medical Capabilities of Gemini
• A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation
• A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
• Extracting Adverse Drug Events from Clinical Notes
• INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis
• MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine
• Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classification
• MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging
• Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model
• MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models

Cited by in corpus (2)

• Differential privacy for medical deep learning: methods, tradeoffs, and deployment implications
• Multi-step retrieval and reasoning improves radiology question answering with large language models