From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine

Abstract

Generative artificial intelligence (AI) models, such as diffusion models and OpenAI's ChatGPT, are transforming medicine by enhancing diagnostic accuracy and automating clinical workflows. The field has advanced rapidly, evolving from text-only large language models for tasks such as clinical documentation and decision support to multimodal AI systems capable of integrating diverse data modalities, including imaging, text, and structured data, within a single model. The diverse landscape of these technologies, along with rising interest, highlights the need for a comprehensive review of their applications and potential. This scoping review explores the evolution of multimodal AI, highlighting its methods, applications, datasets, and evaluation in clinical settings. Adhering to PRISMA-ScR guidelines, we systematically queried PubMed, IEEE Xplore, and Web of Science, prioritizing recent studies published up to the end of 2024. After rigorous screening, 144 papers were included, revealing key trends and challenges in this dynamic field. Our findings underscore a shift from unimodal to multimodal approaches, driving innovations in diagnostic support, medical report generation, drug discovery, and conversational AI. However, critical challenges remain, including the integration of heterogeneous data types, improving model interpretability, addressing ethical concerns, and validating AI systems in real-world clinical settings. This review summarizes the current state of the art, identifies critical gaps, and provides insights to guide the development of scalable, trustworthy, and clinically impactful multimodal AI solutions in healthcare.

References in corpus (13)

• Learning Transferable Visual Models From Natural Language Supervision
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
• Foundation Models in Radiology: What, How, When, Why and Why Not
• GREEN: Generative Radiology Report Evaluation and Error Notation
• Best Practices for Large Language Models in Radiology
• Advancing Multimodal Medical Capabilities of Gemini
• A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
• Extracting Adverse Drug Events from Clinical Notes
• INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis
• MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine
• MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging
• Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model
• MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models

Cited by in corpus (2)

• Differential privacy for medical deep learning: methods, tradeoffs, and deployment implications
• Multi-step retrieval and reasoning improves radiology question answering with large language models