Systematic Generalization: What Is Required and Can It Be Learned?

Abstract

Numerous models for grounded language understanding have been recently proposed, including (i) generic models that can be easily adapted to any given task and (ii) intuitively appealing modular models that require background knowledge to be instantiated. We compare both types of models in how much they lend themselves to a particular form of systematic generalization. Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them. Our findings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected. We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization. We find that end-to-end methods from prior work often learn inappropriate layouts or parametrizations that do not facilitate systematic generalization. Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.

Published as a conference paper at ICLR 2019

References in corpus (6)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Sequence to Sequence Learning with Neural Networks
• Natural Language Inference over Interaction Space
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Differentiable Programs with Neural Libraries
• DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer

Cited by in corpus (13)

• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Environmental drivers of systematicity and generalization in a situated agent
• Compositional generalization through meta sequence-to-sequence learning
• Learning the Travelling Salesperson Problem Requires Rethinking Generalization
• Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
• From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence
• Compositionality decomposed: how do neural networks generalise?
• VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering
• Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• On (Emergent) Systematic Generalisation and Compositionality in Visual Referential Games with Straight-Through Gumbel-Softmax Estimator
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
• Image interpretation by iterative bottom-up top-down processing