What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams

Abstract

Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7\%, 42.0\%, and 70.1\% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.

arXiv commentSubmitted to AAAI 2021

References in corpus (6)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Quasar: Datasets for Question Answering by Search and Reading
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
• Medical Exam Question Answering with Large-scale Reading Comprehension

Cited by in corpus (6)

• A Study of Generative Large Language Model for Medical Research and Healthcare
• Biomedical Question Answering: A Survey of Approaches and Challenges
• OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models
• Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerations
• Evidence Is All You Need: Ordering Imaging Studies via Language Model Alignment with the ACR Appropriateness Criteria
• Recent Advances in Automated Question Answering In Biomedical Domain