Pre-trained Language Model for Biomedical Question Answering

Abstract

The recent success of question answering systems is largely attributed to pre-trained language models. However, as language models are mostly pre-trained on general domain corpora such as Wikipedia, they often have difficulty in understanding biomedical questions. In this paper, we investigate the performance of BioBERT, a pre-trained biomedical language model, in answering biomedical questions including factoid, list, and yes/no type questions. BioBERT uses almost the same structure across various question types and achieved the best performance in the 7th BioASQ Challenge (Task 7b, Phase B). BioBERT pre-trained on SQuAD or SQuAD 2.0 easily outperformed previous state-of-the-art models. BioBERT obtains the best performance when it uses the appropriate pre-/post-processing strategies for questions, passages, and answers.

arXiv commentThis paper is accepted for an oral presentation in BioASQ Workshop @ ECML PKDD 2019

References in corpus (6)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• SciBERT: A Pretrained Language Model for Scientific Text
• Clinical Concept Extraction with Contextual Word Embedding
• Using Neural Networks for Relation Extraction from Biomedical Literature

Cited by in corpus (2)

• Transferability of Natural Language Inference to Biomedical Question Answering
• An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain