ExpMRC: Explainability Evaluation for Machine Reading Comprehension
arXiv:2105.04126 · doi:10.1016/j.heliyon.2022.e09290
Abstract
Achieving human-level performance on some of Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, it is necessary to provide both answer prediction and its explanation to further improve the MRC system's reliability, especially for real-life applications. In this paper, we propose a new benchmark called ExpMRC for evaluating the explainability of the MRC systems. ExpMRC contains four subsets, including SQuAD, CMRC 2018, RACE, and C with additional annotations of the answer's evidence. The MRC systems are required to give not only the correct answer but also its explanation. We use state-of-the-art pre-trained language models to build baseline systems and adopt various unsupervised approaches to extract evidence without a human-annotated training set. The experimental results show that these models are still far from human performance, suggesting that the ExpMRC is challenging. Resources will be available through https://github.com/ymcui/expmrc
arXiv comment10 pages
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Teaching Machines to Read and Comprehend
- Bidirectional Attention Flow for Machine Comprehension
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
- Know What You Don't Know: Unanswerable Questions for SQuAD
- RACE: Large-scale ReAding Comprehension Dataset From Examinations
- Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models