Alignment-based compositional semantics for instruction following

Abstract

This paper describes an alignment-based model for interpreting natural language instructions in context. We approach instruction following as a search over plans, scoring sequences of actions conditioned on structured observations of text and the environment. By explicitly modeling both the low-level compositional structure of individual actions and the high-level structure of full plans, we are able to learn both grounded representations of sentence meaning and pragmatic constraints on interpretation. To demonstrate the model's flexibility, we apply it to a diverse set of benchmark tasks. On every task, we outperform strong task-specific baselines, and achieve several new state-of-the-art results.

in proceedings of EMNLP 2015

References in corpus (2)

• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Compositional Semantic Parsing on Semi-Structured Tables

Cited by in corpus (8)

• Language Understanding for Text-based Games Using Deep Reinforcement Learning
• Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
• Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
• Grounding Language for Transfer in Deep Reinforcement Learning
• Representation Learning for Grounded Spatial Reasoning
• A new dataset and model for learning to understand navigational instructions
• Mapping Natural Language Commands to Web Elements
• From Route Instructions to Landmark Graphs