SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark

Abstract

Existing work in language grounding typically study single environments. How do we build unified models that apply across multiple environments? We propose the multi-environment Symbolic Interactive Language Grounding benchmark (SILG), which unifies a collection of diverse grounded language learning environments under a common interface. SILG consists of grid-world environments that require generalization to new dynamics, entities, and partially observed worlds (RTFM, Messenger, NetHack), as well as symbolic counterparts of visual worlds that require interpreting rich natural language with respect to complex scenes (ALFWorld, Touchdown). Together, these environments provide diverse grounding challenges in richness of observation space, action space, language specification, and plan complexity. In addition, we propose the first shared model architecture for RL on these environments, and evaluate recent advances such as egocentric local convolution, recurrent state-tracking, entity-centric attention, and pretrained LM using SILG. Our shared architecture achieves comparable performance to environment-specific architectures. Moreover, we find that many recent modelling advances do not result in significant gains on environments other than the one they were designed for. This highlights the need for a multi-environment benchmark. Finally, the best models significantly underperform humans on SILG, which suggests ample room for future work. We hope SILG enables the community to quickly identify new methodologies for language grounding that generalize to a diverse set of environments and their associated challenges.

arXiv commentNeurIPS 2021. 14 pages, 8 figures

References in corpus (19)

• The Cityscapes Dataset for Semantic Urban Scene Understanding
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Modular Multitask Reinforcement Learning with Policy Sketches
• dm_control: Software and Tasks for Continuous Control
• Grounded Language Learning in a Simulated 3D World
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Investigating Human Priors for Playing Video Games
• A Benchmark for Systematic Generalization in Grounded Language Understanding
• Learning to Understand Goal Specifications by Modelling Reward
• The StreetLearn Environment and Dataset
• The NetHack Learning Environment
• Grounded Language Learning Fast and Slow
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• TorchBeast: A PyTorch Platform for Distributed RL
• Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight
• Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View
• RTFM: Generalising to Novel Environment Dynamics via Reading
• Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
• MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research