SG-Net: Syntax-Guided Machine Reading Comprehension

Abstract

For machine reading comprehension, the capacity of effectively modeling the linguistic knowledge from the detail-riddled and lengthy passages and getting ride of the noises is essential to improve its performance. Traditional attentive models attend to all words without explicit constraint, which results in inaccurate concentration on some dispensable words. In this work, we propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanism for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the SAN from the original Transformer encoder through a dual contextual architecture for better linguistics inspired representation. To verify its effectiveness, the proposed SG-Net is applied to typical pre-trained language model BERT which is right based on a Transformer encoder. Extensive experiments on popular benchmarks including SQuAD 2.0 and RACE show that the proposed SG-Net design helps achieve substantial performance improvement over strong baselines.

Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2020)

References in corpus (21)

• Teaching Machines to Read and Comprehend
• Bidirectional Attention Flow for Machine Comprehension
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
• Know What You Don't Know: Unanswerable Questions for SQuAD
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Modeling Multi-turn Conversation with Deep Utterance Aggregation
• Syntax-Directed Attention for Neural Machine Translation
• Option Comparison Network for Multiple-choice Reading Comprehension
• Constituency Parsing with a Self-Attentive Encoder
• U-Net: Machine Reading Comprehension with Unanswerable Questions
• Dual Co-Matching Network for Multi-choice Reading Comprehension
• One-shot Learning for Question-Answering in Gaokao History Challenge
• A Fast Unified Model for Parsing and Sentence Understanding
• Subword-augmented Embedding for Cloze Reading Comprehension
• Read + Verify: Machine Reading Comprehension with Unanswerable Questions
• Stochastic Answer Networks for SQuAD 2.0
• LIMIT-BERT : Linguistic Informed Multi-Task BERT
• Stack-Pointer Networks for Dependency Parsing
• Syntax-aware Neural Semantic Role Labeling with Supertags
• Probing Contextualized Sentence Representations with Visual Awareness

Cited by in corpus (6)

• RikiNet: Reading Wikipedia Pages for Natural Question Answering
• Neural Generators of Sparse Local Linear Models for Achieving both Accuracy and Interpretability
• Probing Contextualized Sentence Representations with Visual Awareness
• MCR-Net: A Multi-Step Co-Interactive Relation Network for Unanswerable Questions on Machine Reading Comprehension
• Leveraging Personal Navigation Assistant Systems Using Automated Social Media Traffic Reporting
• Word Embedding-based Text Processing for Comprehensive Summarization and Distinct Information Extraction