Are you doing what I say? On modalities alignment in ALFRED

Abstract

ALFRED is a recently proposed benchmark that requires a model to complete tasks in simulated house environments specified by instructions in natural language. We hypothesize that key to success is accurately aligning the text modality with visual inputs. Motivated by this, we inspect how well existing models can align these modalities using our proposed intrinsic metric, boundary adherence score (BAS). The results show the previous models are indeed failing to perform proper alignment. To address this issue, we introduce approaches aimed at improving model alignment and demonstrate how improved alignment, improves end task performance.

arXiv commentAccepted by Novel Ideas in Learning-to-Learn through Interaction at EMNLP 2021

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning to Navigate in Complex Environments
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
• Are We There Yet? Learning to Localize in Embodied Instruction Following
• Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following
• Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring