JEC-QA: A Legal-Domain Question Answering Dataset
arXiv:1911.12011
Abstract
We present JEC-QA, the largest question answering dataset in the legal domain, collected from the National Judicial Examination of China. The examination is a comprehensive evaluation of professional skills for legal practitioners. College students are required to pass the examination to be certified as a lawyer or a judge. The dataset is challenging for existing question answering methods, because both retrieving relevant materials and answering questions require the ability of logic reasoning. Due to the high demand of multiple reasoning abilities to answer legal questions, the state-of-the-art models can only achieve about 28% accuracy on JEC-QA, while skilled humans and unskilled humans can reach 81% and 64% accuracy respectively, which indicates a huge gap between humans and machines on this task. We will release JEC-QA and our baselines to help improve the reasoning ability of machine comprehension models. You can access the dataset from http://jecqa.thunlp.org/.
arXiv comment9 pages, 2 figures, 10 tables, accepted by AAAI2020
References in corpus (12)
- Teaching Machines to Read and Comprehend
- Bidirectional Attention Flow for Machine Comprehension
- Large-scale Simple Question Answering with Memory Networks
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Machine Comprehension Using Match-LSTM and Answer Pointer
- CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction
- R: Reinforced Reader-Ranker for Open-Domain Question Answering
- NewsQA: A Machine Comprehension Dataset
- Legal Question Answering using Ranking SVM and Deep Convolutional Neural Network
- Convolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions
- A Co-Matching Model for Multi-choice Reading Comprehension
- Medical Exam Question Answering with Large-scale Reading Comprehension