Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
arXiv:2108.04927
Abstract
Language-guided robots performing home and office tasks must navigate in and interact with the world. Grounding language instructions against visual observations and actions to take in an environment is an open challenge. We present Embodied BERT (EmBERT), a transformer-based model which can attend to high-dimensional, multi-modal inputs across long temporal horizons for language-conditioned task completion. Additionally, we bridge the gap between successful object-centric navigation models used for non-interactive agents and the language-guided visual task completion benchmark, ALFRED, by introducing object navigation targets for EmBERT training. We achieve competitive performance on the ALFRED benchmark, and EmBERT marks the first transformer-based model to successfully handle the long-horizon, dense, multi-modal histories of ALFRED, and the first ALFRED model to utilize object-centric navigation targets.
Accepted at Novel Ideas in Learning-to-Learn through Interaction (NILLI) workshop @ EMNLP 2021
References in corpus (21)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Learning Transferable Visual Models From Natural Language Supervision
- Gaussian Error Linear Units (GELUs)
- Longformer: The Long-Document Transformer
- AI2-THOR: An Interactive 3D Environment for Visual AI
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- WikiHow: A Large Scale Text Summarization Dataset
- How multilingual is Multilingual BERT?
- ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects
- Rearrangement: A Challenge for Embodied AI
- Imitating Interactive Intelligence
- Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction
- A Recurrent Vision-and-Language BERT for Navigation
- Sim-to-Real Transfer for Vision-and-Language Navigation
- A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
- Learning Affordance Landscapes for Interaction Exploration in 3D Environments
- Are We There Yet? Learning to Localize in Embodied Instruction Following
- CrossMap Transformer: A Crossmodal Masked Path Transformer Using Double Back-Translation for Vision-and-Language Navigation
- Modular Framework for Visuomotor Language Grounding
- Visually Grounded Continual Learning of Compositional Phrases
- The Amazing Race TM: Robot Edition