On Tree-Based Neural Sentence Modeling
arXiv:1808.09644
Abstract
Neural networks with tree-based sentence encoders have shown better results on many downstream tasks. Most of existing tree-based encoders adopt syntactic parsing trees as the explicit structure prior. To study the effectiveness of different tree structures, we replace the parsing trees with trivial trees (i.e., binary balanced tree, left-branching tree and right-branching tree) in the encoders. Though trivial trees contain no syntactic information, those encoders get competitive or even better results on all of the ten downstream tasks we investigated. This surprising result indicates that explicit syntax guidance may not be the main contributor to the superior performances of tree-based neural sentence modeling. Further analysis show that tree modeling gives better results when crucial words are closer to the final representation. Additional experiments give more clues on how to design an effective tree-based encoder. Our code is open-source and available at https://github.com/ExplorerFreda/TreeEnc.
arXiv commentTo Appear at EMNLP 2018
References in corpus (16)
- Sequence to Sequence Learning with Neural Networks
- Categorical Reparameterization with Gumbel-Softmax
- A Structured Self-attentive Sentence Embedding
- A large annotated corpus for learning natural language inference
- Attentive Pooling Networks
- Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
- Modeling Coverage for Neural Machine Translation
- When Are Tree Structures Necessary for Deep Learning of Representations?
- Bilateral Multi-Perspective Matching for Natural Language Sentences
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
- Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
- Learning to Compose Words into Sentences with Reinforcement Learning
- Tree-to-Sequence Attentional Neural Machine Translation
- Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder
- Modeling Source Syntax for Neural Machine Translation
- Paraphrase Generation with Deep Reinforcement Learning