164 citations · 519 across the 28 of their papers we have counts for
33 papers
MOMASPLAT: Dynamic Novel-View Demonstration Augmentation for Mobile Manipulation
Sateesh Kumar, Arnav Balaji, Arpit Bahety +3
Visuomotor policies trained through imitation learning are highly sensitive to changes in camera viewpoint. This is particularly problematic for mobile manipulation (MoMa), where r…
COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learning
Sateesh Kumar, Shivin Dass, Georgios Pavlakos +1
In this work, we study the problem of data retrieval for few-shot imitation learning: selecting data from a large dataset to train a performant policy for a specific task, given on…
Mixed-Initiative Dialog for Human-Robot Collaborative Manipulation
Albert Yu, Chengshu Li, Luca Macesanu +4
Effective robotic systems for long-horizon human-robot collaboration must adapt to a wide range of human partners, whose physical behavior, willingness to assist, and understanding…
Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models
Huihan Liu, Rutav Shah, Shuijing Liu +6
Assistive teleoperation, where control is shared between a human and a robot, enables efficient and intuitive human-robot collaboration in diverse and unstructured environments. A…
SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation
Arpit Bahety, Arnav Balaji, Ben Abbatematteo +1
For robots to become efficient helpers in the home, they must learn to perform new mobile manipulation tasks simply by watching humans perform them. Learning from a single video de…
SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Training
Jiaheng Hu, Peter Stone, Roberto Martín-Martín
Building capable household and industrial robots requires mastering the control of versatile, high-degree-of-freedom (DoF) systems such as mobile manipulators. While reinforcement…
DataMIL: Selecting Data for Robot Imitation Learning with Datamodels
Shivin Dass, Alaa Khaddaj, Logan Engstrom +3
Recently, the robotics community has amassed ever larger and more diverse datasets to train generalist policies. However, while these policies achieve strong mean performance acros…
Why Automate This? Exploring Correlations Between Desire for Robotic Automation, Invested Time and Well-Being
Ruchira Ray, Leona Pang, Sanjana Srivastava +3
Understanding the motivations underlying the human inclination to automate tasks is vital for developing robots that fit seamlessly into daily life. Accordingly, we ask: are indivi…
BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulation
Rutav Shah, Albert Yu, Yifeng Zhu +2
To operate at a building scale, service robots must perform very long-horizon mobile manipulation tasks by navigating to different rooms, accessing different floors, and interactin…
BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
Yunhao Ge, Yihe Tang, Jiashu Xu +20
The systematic evaluation and understanding of computer vision models under varying conditions require large amounts of data with comprehensive and customized labels, which real-wo…
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
Alexander Khazatsky, Karl Pertsch, Suraj Nair +98
The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and robust robotic manipulation policies. Ho…
Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI
Emily Jin, Jiaheng Hu, Zhuoyi Huang +4
We present Mini-BEHAVIOR, a novel benchmark for embodied AI that challenges agents to use reasoning and decision-making skills to solve complex activities that resemble everyday hu…
MUTEX: Learning Unified Policies from Multimodal Task Specifications
Rutav Shah, Roberto Martín-Martín, Yuke Zhu
Humans use different modalities, such as speech, text, images, videos, etc., to communicate their intent and goals with teammates. For robots to become better assistants, we aim to…
Task-Driven Graph Attention for Hierarchical Relational Object Navigation
Michael Lingelbach, Chengshu Li, Minjune Hwang +6
Embodied AI agents in large scenes often need to navigate to find objects. In this work, we study a naturally emerging variant of the object navigation task, hierarchical relationa…
ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
Le Xue, Ning Yu, Shu Zhang +8
Recent advancements in multimodal pre-training have shown promising efficacy in 3D representation learning by aligning multimodal features across 3D shapes, their 2D counterparts,…
Procedure-Aware Pretraining for Instructional Video Understanding
Honglu Zhou, Roberto Martín-Martín, Mubbasir Kapadia +2
Our goal is to learn a video representation that is useful for downstream procedure understanding tasks in instructional videos. Due to the small amount of available annotations, a…
Hierarchical Point Attention for Indoor 3D Object Detection
Manli Shu, Le Xue, Ning Yu +5
3D object detection is an essential vision technique for various robotic systems, such as augmented reality and domestic robots. Transformers as versatile network architectures hav…
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
Le Xue, Mingfei Gao, Chen Xing +6
The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D coun…
Retrospectives on the Embodied AI Workshop
Matt Deitke, Dhruv Batra, Yonatan Bisk +36
We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped i…
BEHAVIOR in Habitat 2.0: Simulator-Independent Logical Task Description for Benchmarking Embodied AI Agents
Ziang Liu, Roberto Martín-Martín, Fei Xia +2
Robots excel in performing repetitive and precision-sensitive tasks in controlled environments such as warehouses and factories, but have not been yet extended to embodied AI agent…
MaskViT: Masked Visual Pre-Training for Video Prediction
Agrim Gupta, Stephen Tian, Yunzhi Zhang +3
The ability to predict future visual observations conditioned on past observations and motor commands can enable embodied agents to plan solutions to a variety of tasks in complex…
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Ajay Mandlekar, Danfei Xu, Josiah Wong +7
Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch…
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
Sanjana Srivastava, Chengshu Li, Michael Lingelbach +11
We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparat…
iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks
Chengshu Li, Fei Xia, Roberto Martín-Martín +13
Recent research in embodied AI has been boosted by the use of simulation environments to develop and train robot learning approaches. However, the use of simulation has skewed the…