ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
arXiv:2006.12146
Abstract
This paper presents the ReCO, a human-curated ChineseReading Comprehension dataset on Opinion. The questions in ReCO are opinion based queries issued to the commercial search engine. The passages are provided by the crowdworkers who extract the support snippet from the retrieved documents. Finally, an abstractive yes/no/uncertain answer was given by the crowdworkers. The release of ReCO consists of 300k questions that to our knowledge is the largest in Chinese reading comprehension. A prominent characteristic of ReCO is that in addition to the original context paragraph, we also provided the support evidence that could be directly used to answer the question. Quality analysis demonstrates the challenge of ReCO that requires various types of reasoning skills, such as causal inference, logical reasoning, etc. Current QA models that perform very well on many question answering problems, such as BERT, only achieve 77% accuracy on this dataset, a large margin behind humans nearly 92% performance, indicating ReCO presents a good challenge for machine reading comprehension. The codes, datasets are freely available at https://github.com/benywon/ReCO.
arXiv commentAAAI-2020 camera ready
References in corpus (12)
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- A large annotated corpus for learning natural language inference
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- Adversarial Examples for Evaluating Reading Comprehension Systems
- Know What You Don't Know: Unanswerable Questions for SQuAD
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- A BERT Baseline for the Natural Questions
- DRCD: a Chinese Machine Reading Comprehension Dataset
- MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
- Making Neural QA as Simple as Possible but not Simpler
- Efficient and Robust Question Answering from Minimal Context over Documents
- An Attentive Sequence Model for Adverse Drug Event Extraction from Biomedical Text