Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives

Abstract

This paper tackles the problem of reading comprehension over long narratives where documents easily span over thousands of tokens. We propose a curriculum learning (CL) based Pointer-Generator framework for reading/sampling over large documents, enabling diverse training of the neural model based on the notion of alternating contextual difficulty. This can be interpreted as a form of domain randomization and/or generative pretraining during training. To this end, the usage of the Pointer-Generator softens the requirement of having the answer within the context, enabling us to construct diverse training samples for learning. Additionally, we propose a new Introspective Alignment Layer (IAL), which reasons over decomposed alignments using block-based self-attention. We evaluate our proposed method on the NarrativeQA reading comprehension benchmark, achieving state-of-the-art performance, improving existing baselines by $51\%$ relative improvement on BLEU-4 and $17\%$ relative improvement on Rouge-L. Extensive ablations confirm the effectiveness of our proposed IAL and CL components.

Accepted to ACL 2019

References in corpus (18)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ADADELTA: An Adaptive Learning Rate Method
• Teaching Machines to Read and Comprehend
• A Deep Reinforced Model for Abstractive Summarization
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Dynamic Coattention Networks For Question Answering
• Machine Comprehension Using Match-LSTM and Answer Pointer
• The Natural Language Decathlon: Multitask Learning as Question Answering
• An Actor-Critic Algorithm for Sequence Prediction
• Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering
• Get To The Point: Summarization with Pointer-Generator Networks
• Reading Wikipedia to Answer Open-Domain Questions
• NewsQA: A Machine Comprehension Dataset
• S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
• Densely Connected Attention Propagation for Reading Comprehension
• Efficient and Robust Question Answering from Minimal Context over Documents
• Recurrently Controlled Recurrent Networks
• Attention-Guided Answer Distillation for Machine Reading Comprehension

Cited by in corpus (3)

• Sparse Sinkhorn Attention
• Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
• A Survey on Curriculum Learning