6.2k citations · 14.2k across the 134 of their papers we have counts for
163 papers
LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model
Tao Sun, Oliver Liu, JinJin Li +1
Multimodal generative AI usually involves generating image or text responses given inputs in another modality. The evaluation of image-text relevancy is essential for measuring res…
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Xavier Puig, Eric Undersander, Andrew Szot +20
We present Habitat 3.0: a simulation platform for studying collaborative human-robot tasks in home environments. Habitat 3.0 offers contributions across three dimensions: (1) Accur…
What do we learn from a large-scale study of pre-trained visual representations in sim and real environments?
Sneha Silwal, Karmesh Yadav, Tingfan Wu +10
We present a large empirical investigation on the use of pre-trained visual representations (PVRs) for training downstream policies that execute real-world tasks. Our study involve…
Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
Nirbhay Modhe, Qiaozi Gao, Ashwin Kalyan +3
Offline reinforcement learning (RL) methods strike a balance between exploration and exploitation by conservative value estimation -- penalizing values of unseen states and actions…
Skill Transformer: A Monolithic Policy for Mobile Manipulation
Xiaoyu Huang, Dhruv Batra, Akshara Rai +1
We present Skill Transformer, an approach for solving long-horizon robotic tasks by combining conditional sequence modeling and skill modularity. Conditioned on egocentric and prop…
Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Second
Vincent-Pierre Berges, Andrew Szot, Devendra Singh Chaplot +4
We present Galactic, a large-scale simulation and reinforcement-learning (RL) framework for robotic mobile manipulation in indoor environments. Specifically, a Fetch robot (equippe…
HomeRobot: Open-Vocabulary Mobile Manipulation
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav +15
HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks. Open-Vocabulary Mobile Manipulatio…
Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation
Mukul Khanna, Yongsen Mao, Hanxiao Jiang +7
We contribute the Habitat Synthetic Scene Dataset, a dataset of 211 high-quality 3D scenes, and use it to test navigation agent generalization to realistic 3D environments. Our dat…
Adaptive Coordination in Social Embodied Rearrangement
Andrew Szot, Unnat Jain, Dhruv Batra +3
We present the task of "Social Rearrangement", consisting of cooperative everyday tasks like setting up the dinner table, tidying a house or unpacking groceries in a simulated mult…
IndoorSim-to-OutdoorReal: Learning to Navigate Outdoors without any Outdoor Experience
Joanne Truong, April Zitkovich, Sonia Chernova +4
We present IndoorSim-to-OutdoorReal (I2O), an end-to-end learned visual navigation approach, trained solely in simulated short-range indoor environments, and demonstrates zero-shot…
Navigating to Objects Specified by Images
Jacob Krantz, Theophile Gervet, Karmesh Yadav +7
Images are a convenient way to specify which particular object instance an embodied agent should navigate to. Solving this task requires semantic visual reasoning and exploration o…
AutoNeRF: Training Implicit Scene Representations with Autonomous Agents
Pierre Marza, Laetitia Matignon, Olivier Simonin +3
Implicit representations such as Neural Radiance Fields (NeRF) have been shown to be very effective at novel view synthesis. However, these models typically require manual and care…
ASC: Adaptive Skill Coordination for Robotic Mobile Manipulation
Naoki Yokoyama, Alex Clegg, Joanne Truong +6
We present Adaptive Skill Coordination (ASC) -- an approach for accomplishing long-horizon tasks like mobile pick-and-place (i.e., navigating to an object, picking it, navigating t…
BC-IRL: Learning Generalizable Reward Functions from Demonstrations
Andrew Szot, Amy Zhang, Dhruv Batra +2
How well do reward functions learned with inverse reinforcement learning (IRL) generalize? We illustrate that state-of-the-art IRL algorithms, which maximize a maximum-entropy obje…
OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNav
Karmesh Yadav, Arjun Majumdar, Ram Ramrakhya +5
We present a single neural network architecture composed of task-agnostic components (ViTs, convolutions, and LSTMs) that achieves state-of-art results on both the ImageNav ("go to…
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
Arjun Majumdar, Karmesh Yadav, Sergio Arnaud +12
We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBen…
Emergence of Maps in the Memories of Blind Navigation Agents
Erik Wijmans, Manolis Savva, Irfan Essa +3
Animal navigation research posits that organisms build and maintain internal spatial representations, or maps, of their environment. We ask if machines -- specifically, artificial…
PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav
Ram Ramrakhya, Dhruv Batra, Erik Wijmans +1
We study ObjectGoal Navigation -- where a virtual robot situated in a new environment is asked to navigate to an object. Prior work has shown that imitation learning (IL) using beh…
Cross-Domain Transfer via Semantic Skill Imitation
Karl Pertsch, Ruta Desai, Vikash Kumar +4
We propose an approach for semantic imitation, which uses demonstrations from a source domain, e.g. human videos, to accelerate reinforcement learning (RL) in a different target do…
Navigating to Objects in the Real World
Theophile Gervet, Soumith Chintala, Dhruv Batra +2
Semantic navigation is necessary to deploy mobile robots in uncontrolled environments like our homes, schools, and hospitals. Many learning-based approaches have been proposed in r…
Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances
Jacob Krantz, Stefan Lee, Jitendra Malik +2
We consider the problem of embodied visual navigation given an image-goal (ImageNav) where an agent is initialized in an unfamiliar environment and tasked with navigating to a loca…
VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement
Erik Wijmans, Irfan Essa, Dhruv Batra
We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environment…
ViNL: Visual Navigation and Locomotion Over Obstacles
Simar Kareer, Naoki Yokoyama, Dhruv Batra +2
We present Visual Navigation and Locomotion over obstacles (ViNL), which enables a quadrupedal robot to navigate unseen apartments while stepping over small obstacles that lie in i…
Retrospectives on the Embodied AI Workshop
Matt Deitke, Dhruv Batra, Yonatan Bisk +36
We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped i…