Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study

Abstract

Recent advancements in open-domain question answering (ODQA), i.e., finding answers from large open-domain corpus like Wikipedia, have led to human-level performance on many datasets. However, progress in QA over book stories (Book QA) lags behind despite its similar task formulation to ODQA. This work provides a comprehensive and quantitative analysis about the difficulty of Book QA: (1) We benchmark the research on the NarrativeQA dataset with extensive experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a $\sim$7\% absolute improvement on Rouge-L. (2) We further analyze the detailed challenges in Book QA through human studies.\footnote{\url{https://github.com/gorov/BookQA}.} Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.

arXiv commentAccepted to TACL

References in corpus (16)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• REALM: Retrieval-Augmented Language Model Pre-Training
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
• Quasar: Datasets for Question Answering by Search and Reading
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Show Your Work: Improved Reporting of Experimental Results
• Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
• A Discrete Hard EM Approach for Weakly Supervised Question Answering
• Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering
• Multi-style Generative Reading Comprehension
• RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering
• TWEETQA: A Social Media Focused Question Answering Dataset
• ReadTwice: Reading Very Large Documents with Memories
• Complementary Evidence Identification in Open-Domain Question Answering