2.4k citations · 5.2k across the 67 of their papers we have counts for
69 papers
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Zhao Mandi, Shreeya Jain, Shuran Song
We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path p…
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Huy Ha, Pete Florence, Shuran Song
We present a framework for robot skill acquisition, which 1) efficiently scale up data generation of language-labelled robot data and 2) effectively distills this data down into a…
RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstruction
Isaac Kasahara, Shubham Agrawal, Selim Engin +3
General scene reconstruction refers to the task of estimating the full 3D geometry and texture of a scene containing previously unseen objects. In many practical applications such…
XSkill: Cross Embodiment Skill Discovery
Mengda Xu, Zhenjia Xu, Cheng Chi +2
Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior. However, directly extracting reusa…
Rearrangement Planning for General Part Assembly
Yulong Li, Andy Zeng, Shuran Song
Most successes in autonomous robotic assembly have been restricted to single target or category. We propose to investigate general part assembly, the task of creating novel target…
REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction
Zeyi Liu, Arpit Bahety, Shuran Song
The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated…
TidyBot: Personalized Robot Assistance with Large Language Models
Jimmy Wu, Rika Antonova, Adam Kan +6
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate persona…
Pick2Place: Task-aware 6DoF Grasp Estimation via Object-Centric Perspective Affordance
Zhanpeng He, Nikhil Chavan-Dafle, Jinwook Huh +2
The choice of a grasp plays a critical role in the success of downstream manipulation tasks. Consider a task of placing an object in a cluttered scene; the majority of possible gra…
DataComp: In search of the next generation of multimodal datasets
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang +31
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model arch…
Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning
Siddharth Singi, Zhanpeng He, Alvin Pan +5
In a Human-in-the-Loop paradigm, a robotic agent is able to act mostly autonomously in solving a task, but can request help from an external expert when needed. However, knowing wh…
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
Cheng Chi, Zhenjia Xu, Siyuan Feng +5
This paper introduces Diffusion Policy, a new way of generating robot behavior by representing a robot's visuomotor policy as a conditional denoising diffusion process. We benchmar…
RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects
Zhenjia Xu, Zhou Xian, Xingyu Lin +4
We introduce RoboNinja, a learning-based cutting system for multi-material objects (i.e., soft objects with rigid cores such as avocados or mangos). In contrast to prior works usin…
Dynamic Grasping with a Learned Meta-Controller
Yinsen Jia, Jingxi Xu, Dinesh Jayaraman +1
Grasping moving objects is a challenging task that requires multiple submodules such as object pose predictor, arm motion planner, etc. Each submodule operates under its own set of…
Cloth Funnels: Canonicalized-Alignment for Multi-Purpose Garment Manipulation
Alper Canberk, Cheng Chi, Huy Ha +4
Automating garment manipulation is challenging due to extremely high variability in object configurations. To reduce this intrinsic variation, we introduce the task of "canonicaliz…
Bag All You Need: Learning a Generalizable Bagging Strategy for Heterogeneous Objects
Arpit Bahety, Shreeya Jain, Huy Ha +5
We introduce a practical robotics solution for the task of heterogeneous bagging, requiring the placement of multiple rigid and deformable objects into a deformable bag. This is a…
ASPiRe:Adaptive Skill Priors for Reinforcement Learning
Mengda Xu, Manuela Veloso, Shuran Song
We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that leverages prior experience to accelerate reinforcement learning. Unlike existing methods that learn a single…
Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild
Jiayi Chen, Mi Yan, Jiazhao Zhang +6
In this work, we tackle the challenging task of jointly tracking hand object pose and reconstructing their shapes from depth point cloud sequences in the wild, given the initial po…
TANDEM3D: Active Tactile Exploration for 3D Object Recognition
Jingxi Xu, Han Lin, Shuran Song +1
Tactile recognition of 3D objects remains a challenging task. Compared to 2D shapes, the complex geometry of 3D surfaces requires richer tactile signals, more dexterous actions, an…
Patching open-vocabulary models by interpolating weights
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre +5
Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks. However, there are still settings where their zero-shot performance is far from optim…
Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models
Huy Ha, Shuran Song
We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs -…
BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment
Zeyi Liu, Zhenjia Xu, Shuran Song
We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual f…
FishGym: A High-Performance Physics-based Simulation Framework for Underwater Robot Learning
Wenji Liu, Kai Bai, Xuming He +3
Bionic underwater robots have demonstrated their superiority in many applications. Yet, training their intelligence for a variety of tasks that mimic the behavior of underwater cre…
DextAIRity: Deformable Manipulation Can be a Breeze
Zhenjia Xu, Cheng Chi, Benjamin Burchfiel +3
This paper introduces DextAIRity, an approach to manipulate deformable objects using active airflow. In contrast to conventional contact-based quasi-static manipulations, DextAIRit…
Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects
Cheng Chi, Benjamin Burchfiel, Eric Cousineau +2
This paper tackles the task of goal-conditioned dynamic manipulation of deformable objects. This task is highly challenging due to its complex dynamics (introduced by object deform…