Pre-trained Language Models in Biomedical Domain: A Systematic Survey
arXiv:2110.05006
Abstract
Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities propose various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.
Accepted in ACM Computing Surveys
References in corpus (52)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Learning Transferable Visual Models From Natural Language Supervision
- Language Models are Few-Shot Learners
- On the Opportunities and Risks of Foundation Models
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Zero-Shot Text-to-Image Generation
- Publicly Available Clinical BERT Embeddings
- True Few-Shot Learning with Language Models
- Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
- Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
- SciFive: a text-to-text transformer model for biomedical literature
- Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2
- Rapidly Bootstrapping a Question Answering Dataset for COVID-19
- Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
- Conceptualized Representation Learning for Chinese Biomedical Text Mining
- Semi-Supervised Variational Reasoning for Medical Dialogue Generation
- BinaryBERT: Pushing the Limit of BERT Quantization
- Mitigating Gender Bias in Natural Language Processing: Literature Review
- MatchZoo: A Toolkit for Deep Text Matching
- Attention-based Clinical Note Summarization
- Unsupervised Pre-training for Biomedical Question Answering
- MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
- CO-Search: COVID-19 Information Retrieval with Semantic Search, Question Answering, and Abstractive Summarization
- End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Models
- Sequence tagging for biomedical extractive question answering
- Modeling Protein Using Large-scale Pretrain Language Model
- ELECTRAMed: a new pre-trained language representation model for biomedical NLP
- End-to-End QA on COVID-19: Domain Adaptation with Synthetic Training
- Transformer-Based Models for Question Answering on COVID19
- Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
- Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models
- Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models
- Multi-Stage Pre-training for Low-Resource Domain Adaptation
- Combining pre-trained language models and structured knowledge
- BioMegatron: Larger Biomedical Domain Language Model
- Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
- TernaryBERT: Distillation-aware Ultra-low Bit BERT
- On the Generation of Medical Dialogues for COVID-19
- Biomedical named entity recognition using BERT in the machine reading comprehension framework
- Improving Biomedical Pretrained Language Models with Knowledge
- MS2: Multi-Document Summarization of Medical Studies
- Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation
- Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction
- MedGPT: Medical Concept Prediction from Clinical Narratives
- Discourse Probing of Pretrained Language Models
- Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
- Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance
- Experiments on transfer learning architectures for biomedical relation extraction
- Ethics of Artificial Intelligence in Surgery
- Self-Supervised Contextual Language Representation of Radiology Reports to Improve the Identification of Communication Urgency
- Probing Pre-Trained Language Models for Disease Knowledge
- K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce