Tree-structured composition in neural networks without tree-structured architectures

Abstract

Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.

To appear in the proceedings of the 2015 NIPS Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• ADADELTA: An Adaptive Learning Rate Method
• Recurrent Neural Network Regularization
• Visualizing and Understanding Recurrent Networks
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory

Cited by in corpus (8)

• When Are Tree Structures Necessary for Deep Learning of Representations?
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Compositional Generalization by Learning Analytical Expressions
• Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
• The Importance of Being Recurrent for Modeling Hierarchical Structure
• Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons
• A Unifying Framework of Bilinear LSTMs
• Here's My Point: Joint Pointer Architecture for Argument Mining