Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks

Abstract

Neural network models often generalize poorly to mismatched domains or distributions. In NLP, this issue arises in particular when models are expected to generalize compositionally, that is, to novel combinations of familiar words and constructions. We investigate learning representations that facilitate transfer learning from one compositional task to another: the representation and the task-specific layers of the models are strategically trained differently on a pre-finetuning task such that they generalize well on mismatched splits that require compositionality. We apply this method to semantic parsing, using three very different datasets, COGS, GeoQuery and SCAN, used alternately as the pre-finetuning and target task. Our method significantly improves compositional generalization over baselines on the test set of the target task, which is held out during fine-tuning. Ablation studies characterize the utility of the major steps in the proposed algorithm and support our hypothesis.

References in corpus (10)

• An Overview of Multi-Task Learning in Deep Neural Networks
• Theoretical Models of Learning to Learn
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Improving Text-to-SQL Evaluation Methodology
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
• Learning Compositional Rules via Neural Program Synthesis
• Compositional Generalization via Semantic Tagging