Compositional generalization in a deep seq2seq model by separating syntax and semantics

Abstract

Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, human learners readily generalize in this way, e.g. by applying known grammatical rules to novel words. Inspired by work in neuroscience suggesting separate brain systems for syntactic and semantic processing, we implement a modification to standard approaches in neural machine translation, imposing an analogous separation. The novel model, which we call Syntactic Attention, substantially outperforms standard methods in deep learning on the SCAN dataset, a compositional generalization task, without any hand-engineered features or additional supervision. Our work suggests that separating syntactic from semantic learning may be a useful heuristic for capturing compositional structure.

18 pages, 15 figures, preprint version of submission to NeurIPS 2019, under review

References in corpus (2)

• A simple neural network module for relational reasoning
• Deep Learning: A Critical Appraisal

Cited by in corpus (14)

• Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
• Emergent Symbols through Binding in External Memory
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction
• Think before you act: A simple baseline for compositional generalization
• Lexicon Learning for Few-Shot Neural Sequence Modeling
• Analogical Reasoning for Visually Grounded Language Acquisition
• Revisiting Iterative Back-Translation from the Perspective of Compositional Generalization
• Pretrained Embeddings for E-commerce Machine Learning: When it Fails and Why?
• Compositional Neural Machine Translation by Removing the Lexicon from Syntax
• Improving Compositional Generalization in Classification Tasks via Structure Annotations
• ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser
• Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
• Grounded Graph Decoding Improves Compositional Generalization in Question Answering
• Concepts, Properties and an Approach for Compositional Generalization