Annotating Electronic Medical Records for Question Answering

Abstract

Our research is in the relatively unexplored area of question answering technologies for patient-specific questions over their electronic health records. A large dataset of human expert curated question and answer pairs is an important pre-requisite for developing, training and evaluating any question answering system that is powered by machine learning. In this paper, we describe a process for creating such a dataset of questions and answers. Our methodology is replicable, can be conducted by medical students as annotators, and results in high inter-annotator agreement (0.71 Cohen's kappa). Over the course of 11 months, 11 medical students followed our annotation methodology, resulting in a question answering dataset of 5696 questions over 71 patient records, of which 1747 questions have corresponding answers generated by the medical students.

10 pages, 2016

References in corpus (1)

• Teaching Machines to Read and Comprehend

Cited by in corpus (5)

• Biomedical Question Answering: A Survey of Approaches and Challenges
• ANLIzing the Adversarial Natural Language Inference Dataset
• Towards Automatic Generation of Questions from Long Answers
• Development of an Extractive Clinical Question Answering Dataset with Multi-Answer and Multi-Focus Questions
• CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answering