A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
arXiv:2405.08603 · doi:10.1016/j.inffus.2024.102888
Abstract
Since the release of ChatGPT and GPT-4, large language models (LLMs) and multimodal large language models (MLLMs) have attracted widespread attention for their exceptional capabilities in understanding, reasoning, and generation, introducing transformative paradigms for integrating artificial intelligence into medicine. This survey provides a comprehensive overview of the development, principles, application scenarios, challenges, and future directions of LLMs and MLLMs in medicine. Specifically, it begins by examining the paradigm shift, tracing the transition from traditional models to LLMs and MLLMs, and highlighting the unique advantages of these LLMs and MLLMs in medical applications. Next, the survey reviews existing medical LLMs and MLLMs, providing detailed guidance on their construction and evaluation in a clear and systematic manner. Subsequently, to underscore the substantial value of LLMs and MLLMs in healthcare, the survey explores five promising applications in the field. Finally, the survey addresses the challenges confronting medical LLMs and MLLMs and proposes practical strategies and future directions for their integration into medicine. In summary, this survey offers a comprehensive analysis of the technical methodologies and practical clinical applications of medical LLMs and MLLMs, with the goal of bridging the gap between these advanced technologies and clinical practice, thereby fostering the evolution of the next generation of intelligent healthcare systems.
References in corpus (80)
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Survey of Hallucination in Natural Language Generation
- Training language models to follow instructions with human feedback
- LLaMA: Open and Efficient Foundation Language Models
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
- PaLM: Scaling Language Modeling with Pathways
- Flamingo: a Visual Language Model for Few-Shot Learning
- DINOv2: Learning Robust Visual Features without Supervision
- Sequence Level Training with Recurrent Neural Networks
- Gemini: A Family of Highly Capable Multimodal Models
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- A Survey on Multimodal Large Language Models
- Visual Instruction Tuning
- How should my chatbot interact? A survey on human-chatbot interaction design
- Capabilities of GPT-4 on Medical Challenge Problems
- A Study of Generative Large Language Model for Medical Research and Healthcare
- Gender bias and stereotypes in Large Language Models
- Large language models in medicine: the potentials and pitfalls
- Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential
- Towards Expert-Level Medical Question Answering with Large Language Models
- Constitutional AI: Harmlessness from AI Feedback
- GLM-130B: An Open Bilingual Pre-trained Model
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Large AI Models in Health Informatics: Applications, Challenges, and the Future
- LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
- Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
- A Question-Entailment Approach to Question Answering
- Locating and Editing Factual Associations in GPT
- LIMA: Less Is More for Alignment
- MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
- MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data
- A Survey of Hallucination in Large Foundation Models
- Instruction Tuning for Large Language Models: A Survey
- Towards Conversational Diagnostic AI
- HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation
- PathVQA: 30000+ Questions for Medical Visual Question Answering
- Does Synthetic Data Generation of LLMs Help Clinical Text Mining?
- Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and Challenges
- DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
- ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs
- PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
- Deep Learning for Distinguishing Normal versus Abnormal Chest Radiographs and Generalization to Unseen Diseases
- ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation
- ChatCounselor: A Large Language Models for Mental Health Support
- BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
- Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities
- Radiology-Llama2: Best-in-Class Large Language Model for Radiology
- PaLI-3 Vision Language Models: Smaller, Faster, Stronger
- BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT
- Continual Learning for Large Language Models: A Survey
- Data-Centric Foundation Models in Computational Healthcare: A Survey
- M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models
- AlpaCare:Instruction-tuned Large Language Models for Medical Application
- MAIRA-2: Grounded Radiology Report Generation
- Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors
- Towards a Personal Health Large Language Model
- Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model
- Huatuo-26M, a Large-scale Chinese Medical QA Dataset
- OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue
- Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People
- Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare
- Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models
- PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging
- Transformer-Patcher: One Mistake worth One Neuron
- How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making
- Transforming Wearable Data into Personal Health Insights using Large Language Model Agents
- MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine
- MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis
- Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks
- Customizing General-Purpose Foundation Models for Medical Report Generation
- PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning
- MedChatZH: a Better Medical Adviser Learns from Better Instructions
- Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer
- Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models
- DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models