Constituency Parsing with a Self-Attentive Encoder
arXiv:1805.01052
Abstract
We demonstrate that replacing an LSTM encoder with a self-attentive architecture can lead to improvements to a state-of-the-art discriminative constituency parser. The use of attention makes explicit the manner in which information is propagated between different locations in the sentence, which we use to both analyze our model and propose potential improvements. For example, we find that separating positional and content information in the encoder can lead to improved parsing accuracy. Additionally, we evaluate different approaches for lexical representation. Our parser achieves new state-of-the-art results for single models trained on the Penn Treebank: 93.55 F1 without the use of any external data, and 95.13 F1 when using pre-trained word representations. Our parser also outperforms the previous best-published accuracy figures on 8 of the 9 languages in the SPMRL dataset.
arXiv commentACL 2018
References in corpus (1)
Cited by in corpus (23)
- Natural Language Processing Advancements By Deep Learning: A Survey
- Rethinking Positional Encoding in Language Pre-training
- Clinical Concept Extraction with Contextual Word Embedding
- Rethinking Positional Encoding
- SG-Net: Syntax-Guided Machine Reading Comprehension
- TexSmart: A Text Understanding System for Fine-Grained NER and Enhanced Semantic Analysis
- Faster Shift-Reduce Constituent Parsing with a Non-Binary, Bottom-Up Strategy
- Multi-Step Inference for Reasoning Over Paragraphs
- Faithful Multimodal Explanation for Visual Question Answering
- What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure
- Exploiting Temporal Relationships in Video Moment Localization with Natural Language
- Dissecting Contextual Word Embeddings: Architecture and Representation
- Controllable Abstractive Dialogue Summarization with Sketch Supervision
- Discovery of Natural Language Concepts in Individual Units of CNNs
- A Span Selection Model for Semantic Role Labeling
- Graph Ensemble Learning over Multiple Dependency Trees for Aspect-level Sentiment Classification
- Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
- Automated Query Reformulation for Efficient Search based on Query Logs From Stack Overflow
- Direct Output Connection for a High-Rank Language Model
- TENT: Tensorized Encoder Transformer for Temperature Forecasting
- Video Moment Retrieval with Text Query Considering Many-to-Many Correspondence Using Potentially Relevant Pair
- Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing
- Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing