Evaluation of GPT and BERT-based models on identifying protein-protein interactions in biomedical text
arXiv:2303.17728
Abstract
Detecting protein-protein interactions (PPIs) is crucial for understanding genetic mechanisms, disease pathogenesis, and drug design. However, with the fast-paced growth of biomedical literature, there is a growing need for automated and accurate extraction of PPIs to facilitate scientific knowledge discovery. Pre-trained language models, such as generative pre-trained transformers (GPT) and bidirectional encoder representations from transformers (BERT), have shown promising results in natural language processing (NLP) tasks. We evaluated the performance of PPI identification of multiple GPT and BERT models using three manually curated gold-standard corpora: Learning Language in Logic (LLL) with 164 PPIs in 77 sentences, Human Protein Reference Database with 163 PPIs in 145 sentences, and Interaction Extraction Performance Assessment with 335 PPIs in 486 sentences. BERT-based models achieved the best overall performance, with BioBERT achieving the highest recall (91.95%) and F1-score (86.84%) and PubMedBERT achieving the highest precision (85.25%). Interestingly, despite not being explicitly trained for biomedical texts, GPT-4 achieved commendable performance, comparable to the top-performing BERT models. It achieved a precision of 88.37%, a recall of 85.14%, and an F1-score of 86.49% on the LLL dataset. These results suggest that GPT models can effectively detect PPIs from text data, offering promising avenues for application in biomedical literature mining. Further research could explore how these models might be fine-tuned for even more specialized tasks within the biomedical domain.
References in corpus (7)
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
- Large Language Models are Zero-Shot Reasoners
- Capabilities of GPT-4 on Medical Challenge Problems
- The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4
- Construction of the Literature Graph in Semantic Scholar
- Extracting Protein-Protein Interactions (PPIs) from Biomedical Literature using Attention-based Relational Context Information