A Benchmark for Systematic Generalization in Grounded Language Understanding

Abstract

Humans easily interpret expressions that describe unfamiliar situations composed from familiar parts ("greet the pink brontosaurus by the ferris wheel"). Modern neural networks, by contrast, struggle to interpret novel compositions. In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding. Going beyond a related benchmark that focused on syntactic aspects of generalization, gSCAN defines a language grounded in the states of a grid world, facilitating novel evaluations of acquiring linguistically motivated rules. For example, agents must understand how adjectives such as 'small' are interpreted relative to the current world state or how adverbs such as 'cautiously' combine with new verbs. We test a strong multi-modal baseline model and a state-of-the-art compositional method finding that, in most cases, they fail dramatically when generalization requires systematic compositional rules.

arXiv commentaccepted at NeurIPS 2020

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• RobustFill: Neural Program Learning under Noisy I/O
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• ShapeWorld - A new test methodology for multimodal language understanding
• Learning Compositional Rules via Neural Program Synthesis

Cited by in corpus (13)

• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• Compositional Generalization via Neural-Symbolic Stack Machines
• Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
• Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering
• Inducing Causal Structure for Interpretable Neural Networks
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• Zero-Shot Generalization using Intrinsically Motivated Compositional Emergent Protocols
• Robust Generalization of Quadratic Neural Networks via Function Identification
• Concepts, Properties and an Approach for Compositional Generalization
• Compositional Networks Enable Systematic Generalization for Grounded Language Understanding
• ACRE: Abstract Causal REasoning Beyond Covariation
• TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning