MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval

Abstract

Information retrieval (IR) is essential in biomedical knowledge acquisition and clinical decision support. While recent progress has shown that language model encoders perform better semantic retrieval, training such models requires abundant query-article annotations that are difficult to obtain in biomedicine. As a result, most biomedical IR systems only conduct lexical matching. In response, we introduce MedCPT, a first-of-its-kind Contrastively Pre-trained Transformer model for zero-shot semantic IR in biomedicine. To train MedCPT, we collected an unprecedented scale of 255 million user click logs from PubMed. With such data, we use contrastive learning to train a pair of closely-integrated retriever and re-ranker. Experimental results show that MedCPT sets new state-of-the-art performance on six biomedical IR tasks, outperforming various baselines including much larger models such as GPT-3-sized cpt-text-XL. In addition, MedCPT also generates better biomedical article and sentence representations for semantic evaluations. As such, MedCPT can be readily applied to various real-world biomedical IR tasks.

The MedCPT code and API are available at https://github.com/ncbi/MedCPT

Cited by in corpus (11)

• Matching Patients to Clinical Trials with Large Language Models
• MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
• RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements
• Efficient and Reproducible Biomedical Question Answering using Retrieval Augmented Generation
• Leveraging Approximate Caching for Faster Retrieval-Augmented Generation
• Evidence Is All You Need: Ordering Imaging Studies via Language Model Alignment with the ACR Appropriateness Criteria
• MedMobile: A mobile-sized language model with clinical capabilities
• Combining Evidence and Reasoning for Biomedical Fact-Checking
• PlainQAFact: Retrieval-augmented Factual Consistency Evaluation Metric for Biomedical Plain Language Summarization
• ClinicBot: A Guideline-Grounded Clinical Chatbot with Prioritized Evidence RAG and Verifiable Citations
• Harnessing PubMed User Query Logs for Post Hoc Explanations of Recommended Similar Articles