Sequence-to-Sequence Learning with Latent Neural Grammars

Abstract

Sequence-to-sequence learning with neural networks has become the de facto standard for sequence prediction tasks. This approach typically models the local distribution over the next word with a powerful neural network that can condition on arbitrary context. While flexible and performant, these models often require large datasets for training and can fail spectacularly on benchmarks designed to test for compositional generalization. This work explores an alternative, hierarchical approach to sequence-to-sequence learning with quasi-synchronous grammars, where each node in the target tree is transduced by a node in the source tree. Both the source and target trees are treated as latent and induced during training. We develop a neural parameterization of the grammar which enables parameter sharing over the combinatorial space of derivation rules without the need for manual feature engineering. We apply this latent neural grammar to various domains -- a diagnostic language navigation task designed to test for compositional generalization (SCAN), style transfer, and small-scale machine translation -- and find that it performs respectably compared to standard baselines.

arXiv commentNeurIPS 2021

References in corpus (26)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional Sequence to Sequence Learning
• An Efficient Probabilistic Context-Free Parsing Algorithm that Computes Prefix Probabilities
• Online and Linear-Time Attention by Enforcing Monotonic Alignments
• Discovering Discrete Latent Topics with Neural Variational Inference
• Sentence Compression as Tree Transduction
• Segmental Recurrent Neural Networks
• A Retrieve-and-Edit Framework for Predicting Structured Outputs
• Latent Predictor Networks for Code Generation
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Latent Alignment and Variational Attention
• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• Visually Grounded Neural Syntax Acquisition
• Long Short-Term Memory Over Tree Structures
• Compositional Generalization by Learning Analytical Expressions
• The Dynamic Embedded Topic Model
• Towards Neural Phrase-based Machine Translation
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction
• Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence
• StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
• Empirical Study of the Benefits of Overparameterization in Learning Latent Variable Models
• Lexicon Learning for Few-Shot Neural Sequence Modeling
• The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
• Copy that! Editing Sequences by Copying Spans
• Recursive Top-Down Production for Sentence Generation with Latent Trees