Unsupervised Recurrent Neural Network Grammars
arXiv:1904.03746
Abstract
Recurrent neural network grammars (RNNG) are generative models of language which jointly model syntax and surface structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. Supervised RNNGs achieve strong language modeling and parsing performance, but require an annotated corpus of parse trees. In this work, we experiment with unsupervised learning of RNNGs. Since directly marginalizing over the space of latent trees is intractable, we instead apply amortized variational inference. To maximize the evidence lower bound, we develop an inference network parameterized as a neural CRF constituency parser. On language modeling, unsupervised RNNGs perform as well their supervised counterparts on benchmarks in English and Chinese. On constituency grammar induction, they are competitive with recent neural language models that induce tree structures from words through attention mechanisms.
arXiv commentNAACL 2019
References in corpus (12)
- Auto-Encoding Variational Bayes
- Layer Normalization
- Composing graphical models with neural networks for structured representations and fast inference
- Neural Variational Inference and Learning in Belief Networks
- Variational inference for Monte Carlo objectives
- Structured Attention Networks
- Latent Alignment and Variational Attention
- Long Short-Term Memory Over Tree Structures
- Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder
- Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders
- Latent Tree Learning with Differentiable Parsers: Shift-Reduce Parsing and Chart Parsing
- Learning to Discover, Ground and Use Words with Segmental Neural Language Models