306 citations · 404 across the 19 of their papers we have counts for
13 papers · 1 filter
Behavioral Analysis of Vision-and-Language Navigation Agents
Zijiao Yang, Arjun Majumdar, Stefan Lee
To be successful, Vision-and-Language Navigation (VLN) agents must be able to ground instructions to actions based on their surroundings. In this work, we develop a methodology to…
Iterative Vision-and-Language Navigation
Jacob Krantz, Shurjo Banerjee, Wang Zhu +4
We present Iterative Vision-and-Language Navigation (IVLN), a paradigm for evaluating language-guided agents navigating in a persistent environment over time. Existing Vision-and-L…
Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments
Jacob Krantz, Stefan Lee
Recent work in Vision-and-Language Navigation (VLN) has presented two environmental paradigms with differing realism -- the standard VLN setting built on topological environments w…
SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation
Abhinav Moudgil, Arjun Majumdar, Harsh Agrawal +2
Natural language instructions for visual navigation often use scene descriptions (e.g., "bedroom") and object references (e.g., "green chairs") to provide a breadcrumb trail to a g…
Waypoint Models for Instruction-guided Navigation in Continuous Environments
Jacob Krantz, Aaron Gokaslan, Dhruv Batra +2
Little inquiry has explicitly addressed the role of action spaces in language-guided visual navigation -- either in terms of its effect on navigation success or the efficiency with…
Sim-to-Real Transfer for Vision-and-Language Navigation
Peter Anderson, Ayush Shrivastava, Joanne Truong +4
We study the challenging problem of releasing a robot in a previously unseen environment, and having it follow unconstrained natural language navigation instructions. Recent work o…
Where Are You? Localization from Embodied Dialog
Meera Hahn, Jacob Krantz, Dhruv Batra +4
We present Where Are You? (WAY), a dataset of ~6k dialogs in which two humans -- an Observer and a Locator -- complete a cooperative localization task. The Observer is spawned at r…
Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views
Vincent Cartillier, Zhile Ren, Neha Jain +3
We study the task of semantic mapping - specifically, an embodied agent (a robot or an egocentric AI assistant) is given a tour of a new environment and asked to build an allocentr…
Integrating Egocentric Localization for More Realistic Point-Goal Navigation Agents
Samyak Datta, Oleksandr Maksymets, Judy Hoffman +3
Recent work has presented embodied agents that can navigate to point-goal targets in novel indoor environments with near-perfect accuracy. However, these agents are equipped with i…
Dialog without Dialog Data: Learning Visual Dialog Agents from VQA Data
Michael Cogswell, Jiasen Lu, Rishabh Jain +3
Can we develop visually grounded dialog agents that can efficiently adapt to new tasks without forgetting how to talk to people? Such agents could leverage a larger variety of exis…
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web
Arjun Majumdar, Ayush Shrivastava, Stefan Lee +3
Following a navigation instruction such as 'Walk down the stairs and stop at the brown sofa' requires embodied AI agents to ground scene elements referenced via language (e.g. 'sta…
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments
Jacob Krantz, Erik Wijmans, Arjun Majumdar +2
We develop a language-guided navigation task set in a continuous 3D environment where agents must execute low-level actions to follow natural language navigation directions. By bei…
nocaps: novel object captioning at scale
Harsh Agrawal, Karan Desai, Yufei Wang +7
Image captioning models have achieved impressive results on datasets containing limited visual concepts and large amounts of paired image-caption training data. However, if these m…