Structured Reordering for Modeling Latent Alignments in Sequence Transduction
arXiv:2106.03257
Abstract
Despite success in many domains, neural models struggle in settings where train and test examples are drawn from different distributions. In particular, in contrast to humans, conventional sequence-to-sequence (seq2seq) models fail to generalize systematically, i.e., interpret sentences representing novel combinations of concepts (e.g., text segments) seen in training. Traditional grammar formalisms excel in such settings by implicitly encoding alignments between input and output segments, but are hard to scale and maintain. Instead of engineering a grammar, we directly model segment-to-segment alignments as discrete structured latent variables within a neural seq2seq model. To efficiently explore the large space of alignments, we introduce a reorder-first align-later framework whose central component is a neural reordering module producing {\it separable} permutations. We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations, and, thus, enabling end-to-end differentiable training of our model. The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks (i.e., semantic parsing and machine translation).
arXiv commentNeurIPS 2021
References in corpus (16)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Categorical Reparameterization with Gumbel-Softmax
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
- Improving Text-to-SQL Evaluation Methodology
- Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
- Compositional generalization in a deep seq2seq model by separating syntax and semantics
- Latent Alignment and Variational Attention
- Learning to Compose Words into Sentences with Reinforcement Learning
- Towards Neural Phrase-based Machine Translation
- Compositionality decomposed: how do neural networks generalise?
- Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
- Differentiable Ranks and Sorting using Optimal Transport
- Online Segment to Segment Neural Transduction
- Cooperative Learning of Disjoint Syntax and Semantics