Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT

Abstract

We hypothesize that large language models (LLMs) based on the transformer architecture can enable automated detection of clinical phenotype terms, including terms not documented in the HPO. In this study, we developed two types of models: PhenoBCBERT, a BERT-based model, utilizing Bio+Clinical BERT as its pre-trained model, and PhenoGPT, a GPT-based model that can be initialized from diverse GPT models, including open-source versions such as GPT-J, Falcon, and LLaMA, as well as closed-source versions such as GPT-3 and GPT-3.5. We compared our methods with PhenoTagger, a recently developed HPO recognition tool that combines rule-based and deep learning methods. We found that our methods can extract more phenotype concepts, including novel ones not characterized by HPO. We also performed case studies on biomedical literature to illustrate how new phenotype information can be recognized and extracted. We compared current BERT-based versus GPT-based models for phenotype tagging, in multiple aspects including model architecture, memory usage, speed, accuracy, and privacy protection. We also discussed the addition of a negation step and an HPO normalization layer to the transformer models for improved HPO term tagging. In conclusion, PhenoBCBERT and PhenoGPT enable the automated discovery of phenotype terms from clinical notes and biomedical literature, facilitating automated downstream tasks to derive new biological insights on human diseases.

References in corpus (15)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Training language models to follow instructions with human feedback
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Publicly Available Clinical BERT Embeddings
• QLoRA: Efficient Finetuning of Quantized LLMs
• Predictability and Surprise in Large Generative Models
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• BERN2: an advanced neural biomedical named entity recognition and normalization tool
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• NCBO Ontology Recommender 2.0: An Enhanced Approach for Biomedical Ontology Recommendation
• PhenoTagger: A Hybrid Method for Phenotype Concept Recognition using Human Phenotype Ontology

Cited by in corpus (7)

• Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices
• A Hybrid Framework with Large Language Models for Rare Disease Phenotyping
• Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models
• Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes
• High-Throughput Phenotyping of Clinical Text Using Large Language Models
• A Systematic Review on the Generative AI Applications in Human Medical Genomics
• Multimodal Integrated Knowledge Transfer to Large Language Models through Preference Optimization with Biomedical Applications