Recursive Top-Down Production for Sentence Generation with Latent Trees

Abstract

We model the recursive production property of context-free grammars for natural and synthetic languages. To this end, we present a dynamic programming algorithm that marginalises over latent binary tree structures with $N$ leaves, allowing us to compute the likelihood of a sequence of $N$ tokens under a latent tree model, which we maximise to train a recursive neural function. We demonstrate performance on two synthetic tasks: SCAN (Lake and Baroni, 2017), where it outperforms previous models on the LENGTH split, and English question formation (McCoy et al., 2020), where it performs comparably to decoders with the ground-truth tree structure. We also present experimental results on German-English translation on the Multi30k dataset (Elliott et al., 2016), and qualitatively analyse the induced tree structures our model learns for the SCAN tasks and the German-English translation task.

References in corpus (13)

• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• A large annotated corpus for learning natural language inference
• Hierarchical Multiscale Recurrent Neural Networks
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Sequence-Level Knowledge Distillation
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
• Non-Monotonic Sequential Text Generation
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Top-down Tree Long Short-Term Memory Networks
• Unsupervised Recurrent Neural Network Grammars

Cited by in corpus (1)

• Sequence-to-Sequence Learning with Latent Neural Grammars