Pre-trained Language Models in Biomedical Domain: A Systematic Survey
arXiv:2110.05006
Abstract
Pre-trained language models (PLMs) have been the de facto paradigm for most natural language processing (NLP) tasks. This also benefits biomedical domain: researchers from informatics, medicine, and computer science (CS) communities propose various PLMs trained on biomedical datasets, e.g., biomedical text, electronic health records, protein, and DNA sequences for various biomedical tasks. However, the cross-discipline characteristics of biomedical PLMs hinder their spreading among communities; some existing works are isolated from each other without comprehensive comparison and discussions. It expects a survey that not only systematically reviews recent advances of biomedical PLMs and their applications but also standardizes terminology and benchmarks. In this paper, we summarize the recent progress of pre-trained language models in the biomedical domain and their applications in biomedical downstream tasks. Particularly, we discuss the motivations and propose a taxonomy of existing biomedical PLMs. Their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research of the research community.
arXiv commentAccepted in ACM Computing Surveys
References in corpus (83)
- Efficient Estimation of Word Representations in Vector Space
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Learning Transferable Visual Models From Natural Language Supervision
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Language Models are Few-Shot Learners
- On the Opportunities and Risks of Foundation Models
- Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Pre-trained Models for Natural Language Processing: A Survey
- Zero-Shot Text-to-Image Generation
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Publicly Available Clinical BERT Embeddings
- AraBERT: Transformer-based Model for Arabic Language Understanding
- ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
- CORD-19: The COVID-19 Open Research Dataset
- Enhancing Clinical Concept Extraction with Contextual Embeddings
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages
- Learning protein sequence embeddings using information from structure
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
- Evaluation of Text Generation: A Survey
- True Few-Shot Learning with Language Models
- Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
- Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
- SECNLP: A Survey of Embeddings in Clinical Natural Language Processing
- Language Models as Knowledge Bases?
- SciFive: a text-to-text transformer model for biomedical literature
- Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2
- ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
- Encoding word order in complex embeddings
- Rapidly Bootstrapping a Question Answering Dataset for COVID-19
- Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
- Conceptualized Representation Learning for Chinese Biomedical Text Mining
- Semi-Supervised Variational Reasoning for Medical Dialogue Generation
- The Russian Drug Reaction Corpus and Neural Models for Drug Reactions and Effectiveness Detection in User Reviews
- Effective Use of Bidirectional Language Modeling for Transfer Learning in Biomedical Named Entity Recognition
- BinaryBERT: Pushing the Limit of BERT Quantization
- Mitigating Gender Bias in Natural Language Processing: Literature Review
- MatchZoo: A Toolkit for Deep Text Matching
- Attention-based Clinical Note Summarization
- Unsupervised Pre-training for Biomedical Question Answering
- Incorporating Domain Knowledge into Medical NLI using Knowledge Graphs
- MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
- CO-Search: COVID-19 Information Retrieval with Semantic Search, Question Answering, and Abstractive Summarization
- End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Models
- Sequence tagging for biomedical extractive question answering
- Modeling Protein Using Large-scale Pretrain Language Model
- ELECTRAMed: a new pre-trained language representation model for biomedical NLP
- Representation Learning for Electronic Health Records
- End-to-End QA on COVID-19: Domain Adaptation with Synthetic Training
- Transformer-Based Models for Question Answering on COVID19
- Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
- BioFLAIR: Pretrained Pooled Contextualized Embeddings for Biomedical Sequence Labeling Tasks
- Transferability of Natural Language Inference to Biomedical Question Answering
- Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models
- Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models
- Multi-Stage Pre-training for Low-Resource Domain Adaptation
- Combining pre-trained language models and structured knowledge
- BioMegatron: Larger Biomedical Domain Language Model
- TernaryBERT: Distillation-aware Ultra-low Bit BERT
- emrQA: A Large Corpus for Question Answering on Electronic Medical Records
- On the Generation of Medical Dialogues for COVID-19
- Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
- CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management
- Biomedical named entity recognition using BERT in the machine reading comprehension framework
- Learning to Infer Entities, Properties and their Relations from Clinical Conversations
- Improving Biomedical Pretrained Language Models with Knowledge
- Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation
- Continual Domain-Tuning for Pretrained Language Models
- MS2: Multi-Document Summarization of Medical Studies
- Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction
- MedGPT: Medical Concept Prediction from Clinical Narratives
- Discourse Probing of Pretrained Language Models
- Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
- Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance
- Ethics of Artificial Intelligence in Surgery
- Experiments on transfer learning architectures for biomedical relation extraction
- Self-Supervised Contextual Language Representation of Radiology Reports to Improve the Identification of Communication Urgency
- SEntNet: Source-aware Recurrent Entity Network for Dialogue Response Selection
- K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce
- Probing Pre-Trained Language Models for Disease Knowledge
- Clustering of Deep Contextualized Representations for Summarization of Biomedical Texts