Lexicon Learning for Few-Shot Neural Sequence Modeling
arXiv:2106.03993
Abstract
Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following. The neural network models that provide the dominant solution to these problems are brittle, especially in low-resource settings: they fail to generalize correctly or systematically from small datasets. Past work has shown that many failures of systematic generalization arise from neural models' inability to disentangle lexical phenomena from syntactic ones. To address this, we augment neural decoders with a lexical translation mechanism that generalizes existing copy mechanisms to incorporate learned, decontextualized, token-level translation rules. We describe how to initialize this mechanism using a variety of lexicon learning algorithms, and show that it improves systematic generalization on a diverse set of sequence modeling tasks drawn from cognitive science, formal semantics, and machine translation.
arXiv commentACL 2021
References in corpus (6)
- Deep Learning: A Critical Appraisal
- Learning to Transduce with Unbounded Memory
- Compositional generalization in a deep seq2seq model by separating syntax and semantics
- Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
- Pointer-based Fusion of Bilingual Lexicons into Neural Machine Translation
- Compositional Neural Machine Translation by Removing the Lexicon from Syntax