Learning to Compose Words into Sentences with Reinforcement Learning

Abstract

We use reinforcement learning to learn tree-structured neural networks for computing representations of natural language sentences. In contrast with prior work on tree-structured models in which the trees are either provided as input or predicted using supervision from explicit treebank annotations, the tree structures in this work are optimized to improve performance on a downstream task. Experiments demonstrate the benefit of learning task-specific composition orders, outperforming both sequential encoders and recursive encoders based on treebank annotations. We analyze the induced trees and show that while they discover some linguistically intuitive structures (e.g., noun phrases, simple verb phrases), they are different than conventional English syntactic structures.

References in corpus (4)

• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• A Convolutional Neural Network for Modelling Sentences
• A large annotated corpus for learning natural language inference
• Dependency-based Convolutional Neural Networks for Sentence Embedding

Cited by in corpus (22)

• Deep Reinforcement Learning: An Overview
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Neural Models for Information Retrieval
• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• On-the-fly Operation Batching in Dynamic Computation Graphs
• Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction
• BERE: An accurate distantly supervised biomedical entity relation extraction network
• Tensor Product Generation Networks for Deep NLP Modeling
• Direct Optimization through $\arg \max$ for Discrete Variational Auto-Encoder
• Augmenting Neural Networks with First-order Logic
• Inducing Grammars with and for Neural Machine Translation
• Sentence Encoding with Tree-constrained Relation Networks
• Neural Machine Translation with Source-Side Latent Graph Parsing
• Learning to Embed Sentences Using Attentive Recursive Trees
• On Tree-Based Neural Sentence Modeling
• Dynamic Compositionality in Recursive Neural Networks with Structure-aware Tag Representations
• Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces
• Reservoir Stack Machines
• FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding
• Attentive Tensor Product Learning