An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain

Abstract

With the growing amount of text in health data, there have been rapid advances in large pre-trained models that can be applied to a wide variety of biomedical tasks with minimal task-specific modifications. Emphasizing the cost of these models, which renders technical replication challenging, this paper summarizes experiments conducted in replicating BioBERT and further pre-training and careful fine-tuning in the biomedical domain. We also investigate the effectiveness of domain-specific and domain-agnostic pre-trained models across downstream biomedical NLP tasks. Our finding confirms that pre-trained models can be impactful in some downstream NLP tasks (QA and NER) in the biomedical domain; however, this improvement may not justify the high cost of domain-specific pre-training.

References in corpus (10)

• Language Models are Few-Shot Learners
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• End-to-End Open-Domain Question Answering with BERTserini
• FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
• Text Summarization with Pretrained Encoders
• Reproducibility in Machine Learning for Health
• Pre-trained Language Model for Biomedical Question Answering