Compositional generalization in a deep seq2seq model by separating syntax and semantics

Abstract

Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, human learners readily generalize in this way, e.g. by applying known grammatical rules to novel words. Inspired by work in neuroscience suggesting separate brain systems for syntactic and semantic processing, we implement a modification to standard approaches in neural machine translation, imposing an analogous separation. The novel model, which we call Syntactic Attention, substantially outperforms standard methods in deep learning on the SCAN dataset, a compositional generalization task, without any hand-engineered features or additional supervision. Our work suggests that separating syntactic from semantic learning may be a useful heuristic for capturing compositional structure.

18 pages, 15 figures, preprint version of submission to NeurIPS 2019, under review

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• A simple neural network module for relational reasoning
• Deep Learning: A Critical Appraisal
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Compositional Attention Networks for Machine Reasoning
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
• Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

Cited by in corpus (29)

• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• A Benchmark for Systematic Generalization in Grounded Language Understanding
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
• Compositional generalization through meta sequence-to-sequence learning
• CLOSURE: Assessing Systematic Generalization of CLEVR Models
• Compositional Generalization by Learning Analytical Expressions
• Learning Compositional Rules via Neural Program Synthesis
• Hierarchical Poset Decoding for Compositional Generalization in Language
• Emergent Symbols through Binding in External Memory
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction
• When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classification
• Learning to Recombine and Resample Data for Compositional Generalization
• Think before you act: A simple baseline for compositional generalization
• Compositional Generalization via Neural-Symbolic Stack Machines
• Lexicon Learning for Few-Shot Neural Sequence Modeling
• Revisiting Iterative Back-Translation from the Perspective of Compositional Generalization
• Analogical Reasoning for Visually Grounded Language Acquisition
• The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers
• The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
• Pretrained Embeddings for E-commerce Machine Learning: When it Fails and Why?
• Compositional Neural Machine Translation by Removing the Lexicon from Syntax
• ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser
• Improving Compositional Generalization in Classification Tasks via Structure Annotations
• Recursive Top-Down Production for Sentence Generation with Latent Trees
• Grounded Graph Decoding Improves Compositional Generalization in Question Answering
• Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
• Sequence-to-Sequence Learning with Latent Neural Grammars
• Concepts, Properties and an Approach for Compositional Generalization