Machine Reading Comprehension using Case-based Reasoning

Abstract

We present an accurate and interpretable method for answer extraction in machine reading comprehension that is reminiscent of case-based reasoning (CBR) from classical AI. Our method (CBR-MRC) builds upon the hypothesis that contextualized answers to similar questions share semantic similarities with each other. Given a test question, CBR-MRC first retrieves a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers in the retrieved cases. The semi-parametric nature of our approach allows it to attribute a prediction to the specific set of evidence cases, making it a desirable choice for building reliable and debuggable QA systems. We show that CBR-MRC provides high accuracy comparable with large reader models and outperforms baselines by 11.5 and 8.4 EM on NaturalQuestions and NewsQA, respectively. Further, we demonstrate the ability of CBR-MRC in identifying not just the correct answer tokens but also the span with the most relevant supporting evidence. Lastly, we observe that contexts for certain question types show higher lexical diversity than others and find that CBR-MRC is robust to these variations while performance using fully-parametric methods drops.

arXiv comment9 pages, 2 figures

References in corpus (23)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Teaching Machines to Read and Comprehend
• Bidirectional Attention Flow for Machine Comprehension
• Deep Learning for Answer Sentence Selection
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
• Solving Quantitative Reasoning Problems with Language Models
• e-SNLI: Natural Language Inference with Natural Language Explanations
• Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
• What Makes Good In-Context Examples for GPT-$3$?
• Structured information extraction from complex scientific text with fine-tuned large language models
• T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition
• Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
• Compositional Semantic Parsing with Large Language Models
• Binding Language Models in Symbolic Languages
• A Survey on Explainability in Machine Reading Comprehension
• Controllable Semantic Parsing via Retrieval Augmentation
• Efficient and Robust Question Answering from Minimal Context over Documents
• Knowledge Base Question Answering by Case-based Reasoning over Subgraphs
• Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
• Multi-Paragraph Reasoning with Knowledge-enhanced Graph Neural Network
• A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
• CBR-iKB: A Case-Based Reasoning Approach for Question Answering over Incomplete Knowledge Bases