Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond

Abstract

The large-scale development of large language models (LLMs) in medical contexts, such as diagnostic assistance and treatment recommendations, necessitates that these models possess accurate medical knowledge and deliver traceable decision-making processes. Clinical knowledge, encompassing the insights gained from research on the causes, prognosis, diagnosis, and treatment of diseases, has been extensively examined within real-world medical practices. Recently, there has been a notable increase in research efforts aimed at integrating this type of knowledge into LLMs, encompassing not only traditional text and multimodal data integration but also technologies such as knowledge graphs (KGs) and retrieval-augmented generation (RAG). In this paper, we review the various initiatives to embed clinical knowledge into training-based, KG-supported, and RAG-assisted LLMs. We begin by gathering reliable knowledge sources from the medical domain, including databases and datasets. Next, we evaluate implementations for integrating clinical knowledge through specialized datasets and collaborations with external knowledge sources such as KGs and relevant documentation. Furthermore, we discuss the applications of the developed medical LLMs in the industrial sector to assess the disparity between models developed in academic settings and those in industry. We conclude the survey by presenting evaluation systems applicable to relevant tasks and identifying potential challenges facing this field. In this review, we do not aim for completeness, since any ostensibly complete review would soon be outdated. Our goal is to illustrate diversity by selecting representative and accessible items from current research and industry practices, reflecting real-world situations rather than claiming completeness. Thus, we emphasize showcasing diverse approaches.

Accepted for publication in Knowledge-Based Systems. The arXiv version is the pre-peer-review preprint, and the final published version is not available here due to publisher policy

References in corpus (46)

• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• PaLM: Scaling Language Modeling with Pathways
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Publicly Available Clinical BERT Embeddings
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• A Study of Generative Large Language Model for Medical Research and Healthcare
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks
• A Question-Entailment Approach to Question Answering
• GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
• OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
• Instruction Tuning for Large Language Models: A Survey
• HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
• Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study
• PathVQA: 30000+ Questions for Medical Visual Question Answering
• DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
• GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records
• ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation
• BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
• Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation
• Conversational Health Agents: A Personalized LLM-Powered Agent Framework
• HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs
• BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT
• Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs
• Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases
• DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation
• Agents: An Open-source Framework for Autonomous Language Agents
• Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model
• Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT
• Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare
• KG4Diagnosis: A Hierarchical Multi-Agent LLM Framework with Knowledge Graph Enhancement for Medical Diagnosis
• TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
• Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education
• Depression Diagnosis Dialogue Simulation: Self-improving Psychiatrist with Tertiary Memory
• A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction
• MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents
• Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval
• MDTeamGPT: A Self-Evolving LLM-based Multi-Agent Framework for Multi-Disciplinary Team Medical Consultation
• Ziya2: Data-centric Learning is All LLMs Need
• Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators
• Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge
• Stone Needle: A General Multimodal Large-scale Model Framework towards Healthcare
• RJUA-QA: A Comprehensive QA Dataset for Urology