Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
arXiv:2106.13375 · doi:10.1145/3447548.3469053
Abstract
Information overload is a prevalent challenge in many high-value domains. A prominent case in point is the explosion of the biomedical literature on COVID-19, which swelled to hundreds of thousands of papers in a matter of months. In general, biomedical literature expands by two papers every minute, totalling over a million new papers every year. Search in the biomedical realm, and many other vertical domains is challenging due to the scarcity of direct supervision from click logs. Self-supervised learning has emerged as a promising direction to overcome the annotation bottleneck. We propose a general approach for vertical search based on domain-specific pretraining and present a case study for the biomedical domain. Despite being substantially simpler and not using any relevance labels for training or development, our method performs comparably or better than the best systems in the official TREC-COVID evaluation, a COVID-related biomedical search competition. Using distributed computing in modern cloud infrastructure, our system can scale to tens of millions of articles on PubMed and has been deployed as Microsoft Biomedical Search, a new search experience for biomedical literature: https://aka.ms/biomedsearch.
ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2021 Applied Data Science Track
References in corpus (14)
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Language Models are Few-Shot Learners
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
- CORD-19: The COVID-19 Open Research Dataset
- Passage Re-ranking with BERT
- Enhancing Clinical Concept Extraction with Contextual Embeddings
- Simple Applications of BERT for Ad Hoc Document Retrieval
- Pretrained Transformers for Text Ranking: BERT and Beyond
- Selective Weak Supervision for Neural Information Retrieval
- CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
- UnitedQA: A Hybrid Approach for Open Domain Question Answering