FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension

Abstract

This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives. First, it puts forward a novel concept of "history of word" to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation. Second, it introduces an improved attention scoring function that better utilizes the "history of word" concept. Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer. We apply FusionNet to the Stanford Question Answering Dataset (SQuAD) and it achieves the first position for both single and ensemble model on the official SQuAD leaderboard at the time of writing (Oct. 4th, 2017). Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.

arXiv commentPublished in Sixth International Conference on Learning Representations (ICLR), 2018

Cited by in corpus (17)

• Deep Learning Based Text Classification: A Comprehensive Review
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Temporal Sentence Grounding in Videos: A Survey and Future Directions
• Parallel Attention Network with Sequence Matching for Video Grounding
• U-Net: Machine Reading Comprehension with Unanswerable Questions
• Convolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions
• Towards Data Distillation for End-to-end Spoken Conversational Question Answering
• Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
• Dynamic Fusion Networks for Machine Reading Comprehension
• Read + Verify: Machine Reading Comprehension with Unanswerable Questions
• Weaver: Deep Co-Encoding of Questions and Documents for Machine Reading
• Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modeling
• A Vietnamese Dataset for Evaluating Machine Reading Comprehension
• Momentum Contrastive Pre-training for Question Answering
• An Empirical Study of Content Understanding in Conversational Question Answering
• A3Net: Adversarial-and-Attention Network for Machine Reading Comprehension
• Question-Aware Sentence Gating Networks for Question and Answering