Sentence Encoding with Tree-constrained Relation Networks
arXiv:1811.10475
Abstract
The meaning of a sentence is a function of the relations that hold between its words. We instantiate this relational view of semantics in a series of neural models based on variants of relation networks (RNs) which represent a set of objects (for us, words forming a sentence) in terms of representations of pairs of objects. We propose two extensions to the basic RN model for natural language. First, building on the intuition that not all word pairs are equally informative about the meaning of a sentence, we use constraints based on both supervised and unsupervised dependency syntax to control which relations influence the representation. Second, since higher-order relations are poorly captured by a sum of pairwise relations, we use a recurrent extension of RNs to propagate information so as to form representations of higher order relations. Experiments on sentence classification, sentence pair classification, and machine translation reveal that, while basic RNs are only modestly effective for sentence representation, recurrent RNs with latent syntax are a reliably powerful representational device.
12 pages
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Categorical Reparameterization with Gumbel-Softmax
- A simple neural network module for relational reasoning
- A Convolutional Neural Network for Modelling Sentences
- A large annotated corpus for learning natural language inference
- Long Short-Term Memory-Networks for Machine Reading
- Bilateral Multi-Perspective Matching for Natural Language Sentences
- Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
- Dependency-based Convolutional Neural Networks for Sentence Embedding
- Learning to Compose Words into Sentences with Reinforcement Learning
- Structured Training for Neural Network Transition-Based Parsing
- Neural Probabilistic Model for Non-projective MST Parsing
- DRAGNN: A Transition-based Framework for Dynamically Connected Neural Networks
- Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
- Recurrent Relational Networks
- Neural Paraphrase Identification of Questions with Noisy Pretraining
- Learning Structured Text Representations
- Do latent tree learning models identify meaningful structure in sentences?