Understanding Dataset Design Choices for Multi-hop Reasoning

Abstract

Learning multi-hop reasoning has been a key challenge for reading comprehension models, leading to the design of datasets that explicitly focus on it. Ideally, a model should not be able to perform well on a multi-hop question answering task without doing multi-hop reasoning. In this paper, we investigate two recently proposed datasets, WikiHop and HotpotQA. First, we explore sentence-factored models for these tasks; by design, these models cannot do multi-hop reasoning, but they are still able to solve a large number of examples in both datasets. Furthermore, we find spurious correlations in the unmasked version of WikiHop, which make it easy to achieve high performance considering only the questions and answers. Finally, we investigate one key difference between these datasets, namely span-based vs. multiple-choice formulations of the QA task. Multiple-choice versions of both datasets can be easily gamed, and two models we examine only marginally exceed a baseline in this setting. Overall, while these datasets are useful testbeds, high-performing models may not be learning as much multi-hop reasoning as previously thought.

arXiv commentNAACL 2019

References in corpus (5)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Teaching Machines to Read and Comprehend
• Bidirectional Attention Flow for Machine Comprehension
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• ReasoNet: Learning to Stop Reading in Machine Comprehension

Cited by in corpus (2)

• Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
• Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval