Phase Conductor on Multi-layered Attentions for Machine Comprehension

Abstract

Attention models have been intensively studied to improve NLP tasks such as machine comprehension via both question-aware passage attention model and self-matching attention model. Our research proposes phase conductor (PhaseCond) for attention models in two meaningful ways. First, PhaseCond, an architecture of multi-layered attention models, consists of multiple phases each implementing a stack of attention layers producing passage representations and a stack of inner or outer fusion layers regulating the information flow. Second, we extend and improve the dot-product attention function for PhaseCond by simultaneously encoding multiple question and passage embedding layers from different perspectives. We demonstrate the effectiveness of our proposed model PhaseCond on the SQuAD dataset, showing that our model significantly outperforms both state-of-the-art single-layered and multiple-layered attention models. We deepen our results with new findings via both detailed qualitative analysis and visualized examples showing the dynamic changes through multi-layered attention models.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Machine Comprehension Using Match-LSTM and Answer Pointer
• MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
• Reinforced Mnemonic Reader for Machine Reading Comprehension
• Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering

Cited by in corpus (7)

• The Natural Language Decathlon: Multitask Learning as Question Answering
• PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text
• Stochastic Answer Networks for Machine Reading Comprehension
• The Effect of Natural Distribution Shift on Question Answering Models
• Weaver: Deep Co-Encoding of Questions and Documents for Machine Reading
• Making Neural Machine Reading Comprehension Faster
• Question-Aware Sentence Gating Networks for Question and Answering