Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures

Abstract

While mainstream machine learning methods are known to have limited ability to compositionally generalize, new architectures and techniques continue to be proposed to address this limitation. We investigate state-of-the-art techniques and architectures in order to assess their effectiveness in improving compositional generalization in semantic parsing tasks based on the SCAN and CFQ datasets. We show that masked language model (MLM) pre-training rivals SCAN-inspired architectures on primitive holdout splits. On a more complex compositional task, we show that pre-training leads to significant improvements in performance vs. comparable non-pre-trained models, whereas architectures proposed to encourage compositional generalization on SCAN or in the area of algorithm learning fail to lead to significant improvements. We establish a new state of the art on the CFQ compositional generalization benchmark using MLM pre-training together with an intermediate representation.

References in corpus (35)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Neural Message Passing for Quantum Chemistry
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Group Equivariant Convolutional Networks
• Gated Graph Sequence Neural Networks
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Improving Text-to-SQL Evaluation Methodology
• Neural GPUs Learn Algorithms
• The Evolved Transformer
• Order Matters: Sequence to sequence for sets
• Lambda Dependency-Based Compositional Semantics
• A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
• A Syntactic Neural Model for General-Purpose Code Generation
• DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• SCAN: Learning Hierarchical Compositional Visual Concepts
• A Benchmark for Systematic Generalization in Grounded Language Understanding
• Unsupervised Translation of Programming Languages
• IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• Environmental drivers of systematicity and generalization in a situated agent
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
• CLOSURE: Assessing Systematic Generalization of CLEVR Models
• PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text
• Learning Compositional Rules via Neural Program Synthesis
• Compositional Generalization by Learning Analytical Expressions
• Improving the Neural GPU Architecture for Algorithm Learning
• Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation
• Neural Shuffle-Exchange Networks -- Sequence Processing in O(n log n) Time
• Compositional Generalization for Primitive Substitutions
• Global Reasoning over Database Structures for Text-to-SQL Parsing
• Generating Logical Forms from Graph Representations of Text and Entities

Cited by in corpus (16)

• Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
• Modern Baselines for SPARQL Semantic Parsing
• An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
• Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing
• Lexicon Learning for Few-Shot Neural Sequence Modeling
• Revisiting Iterative Back-Translation from the Perspective of Compositional Generalization
• The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization
• Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks
• Improving Compositional Generalization in Classification Tasks via Structure Annotations
• Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
• Sequence-to-Sequence Learning with Latent Neural Grammars
• SQALER: Scaling Question Answering by Decoupling Multi-Hop and Logical Reasoning
• Learning Algebraic Recombination for Compositional Generalization
• LAGr: Labeling Aligned Graphs for Improving Systematic Generalization in Semantic Parsing
• Iterative Decoding for Compositional Generalization in Transformers
• Compositional generalization in semantic parsing with pretrained transformers