ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
arXiv:2011.07660
Abstract
For embodied agents, navigation is an important ability but not an isolated goal. Agents are also expected to perform specific tasks after reaching the target location, such as picking up objects and assembling them into a particular arrangement. We combine Vision-and-Language Navigation, assembling of collected objects, and object referring expression comprehension, to create a novel joint navigation-and-assembly task, named ArraMon. During this task, the agent (similar to a PokeMON GO player) is asked to find and collect different target objects one-by-one by navigating based on natural language instructions in a complex, realistic outdoor environment, but then also ARRAnge the collected objects part-by-part in an egocentric grid-layout environment. To support this task, we implement a 3D dynamic environment simulator and collect a dataset (in English; and also extended to Hindi) with human-written navigation and assembling instructions, and the corresponding ground truth trajectories. We also filter the collected instructions via a verification stage, leading to a total of 7.7K task instances (30.8K instructions and paths). We present results for several baseline models (integrated and biased) and metrics (nDTW, CTC, rPOD, and PTC), and the large model-human performance gap demonstrates that our task is challenging and presents a wide scope for future work. Our dataset, simulator, and code are publicly available at: https://arramonunc.github.io
arXiv commentEMNLP Findings 2020 (18 pages; extended to Hindi)
References in corpus (12)
- AI2-THOR: An Interactive 3D Environment for Visual AI
- MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
- Grounded Language Learning in a Simulated 3D World
- Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
- MAttNet: Modular Attention Network for Referring Expression Comprehension
- HoME: a Household Multimodal Environment
- CHALET: Cornell House Agent Learning Environment
- Learning to Navigate in Cities Without a Map
- Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction
- FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning
- VirtualHome: Simulating Household Activities via Programs
- Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning