Compositional Distributional Semantics with Long Short Term Memory

Abstract

We are proposing an extension of the recursive neural network that makes use of a variant of the long short-term memory architecture. The extension allows information low in parse trees to be stored in a memory register (the `memory cell') and used much later higher up in the parse tree. This provides a solution to the vanishing gradient problem and allows the network to capture long range dependencies. Experimental results show that our composition outperformed the traditional neural-network composition on the Stanford Sentiment Treebank.

10 pages, 7 figures

References in corpus (2)

• A Convolutional Neural Network for Modelling Sentences
• Long Short-Term Memory Over Tree Structures

Cited by in corpus (10)

• CNN-RNN: A Unified Framework for Multi-label Image Classification
• Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings
• Discriminative Neural Sentence Modeling by Tree-Based Convolution
• Bidirectional Tree-Structured LSTM with Head Lexicalization
• Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons
• Quantifying the vanishing gradient and long distance dependency problem in recursive neural networks and recursive LSTMs
• Can Neural Networks Understand Logical Entailment?
• Semantic Word Clusters Using Signed Normalized Graph Cuts
• TanhSoft -- a family of activation functions combining Tanh and Softplus
• Paraphrasing verbal metonymy through computational methods