Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Abstract

Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

arXiv commentPublished as a conference paper at ICLR 2019

References in corpus (22)

• Deep Learning in Neural Networks: An Overview
• Neural Architecture Search with Reinforcement Learning
• Recurrent Neural Network Regularization
• Character-Aware Neural Language Models
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• A Clockwork RNN
• Hierarchical Multiscale Recurrent Neural Networks
• Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Gradient Estimation Using Stochastic Computation Graphs
• Learning to Transduce with Unbounded Memory
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Tree-structured composition in neural networks without tree-structured architectures
• Learning to Compose Words into Sentences with Reinforcement Learning
• Learning Ordered Representations with Nested Dropout
• A Fast Unified Model for Parsing and Sentence Understanding
• Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Targeted Syntactic Evaluation of Language Models
• Top-down Tree Long Short-Term Memory Networks
• On Tree-Based Neural Sentence Modeling

Cited by in corpus (23)

• Deep Neural Approaches to Relation Triplets Extraction: A Comprehensive Survey
• Improving the Gating Mechanism of Recurrent Neural Networks
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Selfish Sparse RNN Training
• Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization
• Can RNNs learn Recursive Nested Subject-Verb Agreements?
• Tree Transformer: Integrating Tree Structures into Self-Attention
• Transformers Generalize Linearly
• Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
• StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
• Structure-Aware Generation Network for Recipe Generation from Images
• Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
• Tha3aroon at NSURL-2019 Task 8: Semantic Question Similarity in Arabic
• NSURL-2019 Shared Task 8: Semantic Question Similarity in Arabic
• Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?
• Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach
• Recursive Top-Down Production for Sentence Generation with Latent Trees
• Learning Algebraic Recombination for Compositional Generalization
• Semi-supervised Autoencoding Projective Dependency Parsing
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding
• Joint Embedding of Words and Category Labels for Hierarchical Multi-label Text Classification
• Unique Chinese Linguistic Phenomena