19 citations · 79 across the 16 of their papers we have counts for
17 papers
Pixtral 12B
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna +39
We introduce Pixtral-12B, a 12--billion-parameter multimodal language model. Pixtral-12B is trained to understand both natural images and documents, achieving leading performance o…
Situated Instruction Following
So Yeon Min, Xavi Puig, Devendra Singh Chaplot +6
Language is never spoken in a vacuum. It is expressed, comprehended, and contextualized within the holistic backdrop of the speaker's history, actions, and environment. Since human…
Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge
Sriram Yenamandra, Arun Ramachandran, Mukul Khanna +42
In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of obje…
GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation
Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani +7
The Embodied AI community has made significant strides in visual navigation tasks, exploring targets from 3D coordinates, objects, language descriptions, and images. However, these…
GOAT: GO to Any Thing
Matthew Chang, Theophile Gervet, Mukul Khanna +10
In deployment scenarios such as homes and warehouses, mobile robots are expected to autonomously navigate for extended periods, seamlessly executing tasks articulated in terms that…
Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at 100k Steps-Per-Second
Vincent-Pierre Berges, Andrew Szot, Devendra Singh Chaplot +4
We present Galactic, a large-scale simulation and reinforcement-learning (RL) framework for robotic mobile manipulation in indoor environments. Specifically, a Fetch robot (equippe…
HomeRobot: Open-Vocabulary Mobile Manipulation
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav +15
HomeRobot (noun): An affordable compliant robot that navigates homes and manipulates a wide range of objects in order to complete everyday tasks. Open-Vocabulary Mobile Manipulatio…
Navigating to Objects Specified by Images
Jacob Krantz, Theophile Gervet, Karmesh Yadav +7
Images are a convenient way to specify which particular object instance an embodied agent should navigate to. Solving this task requires semantic visual reasoning and exploration o…
AutoNeRF: Training Implicit Scene Representations with Autonomous Agents
Pierre Marza, Laetitia Matignon, Olivier Simonin +3
Implicit representations such as Neural Radiance Fields (NeRF) have been shown to be very effective at novel view synthesis. However, these models typically require manual and care…
Navigating to Objects in the Real World
Theophile Gervet, Soumith Chintala, Dhruv Batra +2
Semantic navigation is necessary to deploy mobile robots in uncontrolled environments like our homes, schools, and hospitals. Many learning-based approaches have been proposed in r…
Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances
Jacob Krantz, Stefan Lee, Jitendra Malik +2
We consider the problem of embodied visual navigation given an image-goal (ImageNav) where an agent is initialized in an unfamiliar environment and tasked with navigating to a loca…
Retrospectives on the Embodied AI Workshop
Matt Deitke, Dhruv Batra, Yonatan Bisk +36
We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped i…
Multi-skill Mobile Manipulation for Object Rearrangement
Jiayuan Gu, Devendra Singh Chaplot, Hao Su +1
We study a modular approach to tackle long-horizon mobile manipulation tasks for object rearrangement, which decomposes a full task into a sequence of subtasks. To tackle the entir…
PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning
Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah +2
State-of-the-art approaches to ObjectGoal navigation rely on reinforcement learning and typically require significant computational resources and time for learning. We propose Pote…
Recognizing Scenes from Novel Viewpoints
Shengyi Qian, Alexander Kirillov, Nikhila Ravi +4
Humans can perceive scenes in 3D from a handful of 2D views. For AI agents, the ability to recognize a scene from any viewpoint given only a few images enables them to efficiently…
SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency
Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta +2
In this paper, we explore how we can build upon the data and models of Internet images and use them to adapt to robot vision without requiring any extra labels. We present a framew…
Embodied Multimodal Multitask Learning
Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov +2
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tas…