Towards Inference-Oriented Reading Comprehension: ParallelQA

Abstract

In this paper, we investigate the tendency of end-to-end neural Machine Reading Comprehension (MRC) models to match shallow patterns rather than perform inference-oriented reasoning on RC benchmarks. We aim to test the ability of these systems to answer questions which focus on referential inference. We propose ParallelQA, a strategy to formulate such questions using parallel passages. We also demonstrate that existing neural models fail to generalize well to this setting.

Accepted at Workshop on New Forms of Generalization in Deep Learning and Natural Language Processing, NAACL 2018