Neural Language Modeling by Jointly Learning Syntax and Lexicon

Abstract

We propose a neural language model capable of unsupervised syntactic structure induction. The model leverages the structure information to form better semantic representations and better language modeling. Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information. On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation. In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network. Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks.

16 pages, 5 figures, ICLR 2018

References in corpus (21)

• Efficient Estimation of Word Representations in Vector Space
• Deep Learning in Neural Networks: An Overview
• Neural Architecture Search with Reinforcement Learning
• Recurrent Neural Network Regularization
• Character-Aware Neural Language Models
• Pointer Sentinel Mixture Models
• A Clockwork RNN
• Hierarchical Multiscale Recurrent Neural Networks
• Learning Longer Memory in Recurrent Neural Networks
• Long Short-Term Memory-Networks for Machine Reading
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• The Expressive Power of Word Embeddings
• On Multiplicative Integration with Recurrent Neural Networks
• Recurrent Batch Normalization
• A Fast Unified Model for Parsing and Sentence Understanding
• Regularizing RNNs by Stabilizing Activations
• What Do Recurrent Neural Network Grammars Learn About Syntax?
• Top-down Tree Long Short-Term Memory Networks
• Do latent tree learning models identify meaningful structure in sentences?
• Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation

Cited by in corpus (11)

• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• A Comprehensive Survey of Grammar Error Correction
• Structure-Aware Generation Network for Recipe Generation from Images
• Straight to the Tree: Constituency Parsing with Neural Syntactic Distance
• Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach
• Recursive Top-Down Production for Sentence Generation with Latent Trees
• FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
• Learning Noun Cases Using Sequential Neural Networks