2.4k citations · 5.2k across the 67 of their papers we have counts for
36 papers · 1 filter
RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
Zhao Mandi, Shreeya Jain, Shuran Song
We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path p…
Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
Huy Ha, Pete Florence, Shuran Song
We present a framework for robot skill acquisition, which 1) efficiently scale up data generation of language-labelled robot data and 2) effectively distills this data down into a…
XSkill: Cross Embodiment Skill Discovery
Mengda Xu, Zhenjia Xu, Cheng Chi +2
Human demonstration videos are a widely available data source for robot learning and an intuitive user interface for expressing desired behavior. However, directly extracting reusa…
Rearrangement Planning for General Part Assembly
Yulong Li, Andy Zeng, Shuran Song
Most successes in autonomous robotic assembly have been restricted to single target or category. We propose to investigate general part assembly, the task of creating novel target…
REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction
Zeyi Liu, Arpit Bahety, Shuran Song
The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated…
TidyBot: Personalized Robot Assistance with Large Language Models
Jimmy Wu, Rika Antonova, Adam Kan +6
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate persona…
Pick2Place: Task-aware 6DoF Grasp Estimation via Object-Centric Perspective Affordance
Zhanpeng He, Nikhil Chavan-Dafle, Jinwook Huh +2
The choice of a grasp plays a critical role in the success of downstream manipulation tasks. Consider a task of placing an object in a cluttered scene; the majority of possible gra…
Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning
Siddharth Singi, Zhanpeng He, Alvin Pan +5
In a Human-in-the-Loop paradigm, a robotic agent is able to act mostly autonomously in solving a task, but can request help from an external expert when needed. However, knowing wh…
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
Cheng Chi, Zhenjia Xu, Siyuan Feng +5
This paper introduces Diffusion Policy, a new way of generating robot behavior by representing a robot's visuomotor policy as a conditional denoising diffusion process. We benchmar…
RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects
Zhenjia Xu, Zhou Xian, Xingyu Lin +4
We introduce RoboNinja, a learning-based cutting system for multi-material objects (i.e., soft objects with rigid cores such as avocados or mangos). In contrast to prior works usin…
Dynamic Grasping with a Learned Meta-Controller
Yinsen Jia, Jingxi Xu, Dinesh Jayaraman +1
Grasping moving objects is a challenging task that requires multiple submodules such as object pose predictor, arm motion planner, etc. Each submodule operates under its own set of…
Cloth Funnels: Canonicalized-Alignment for Multi-Purpose Garment Manipulation
Alper Canberk, Cheng Chi, Huy Ha +4
Automating garment manipulation is challenging due to extremely high variability in object configurations. To reduce this intrinsic variation, we introduce the task of "canonicaliz…
Bag All You Need: Learning a Generalizable Bagging Strategy for Heterogeneous Objects
Arpit Bahety, Shreeya Jain, Huy Ha +5
We introduce a practical robotics solution for the task of heterogeneous bagging, requiring the placement of multiple rigid and deformable objects into a deformable bag. This is a…
BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment
Zeyi Liu, Zhenjia Xu, Shuran Song
We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual f…
FishGym: A High-Performance Physics-based Simulation Framework for Underwater Robot Learning
Wenji Liu, Kai Bai, Xuming He +3
Bionic underwater robots have demonstrated their superiority in many applications. Yet, training their intelligence for a variety of tasks that mimic the behavior of underwater cre…
DextAIRity: Deformable Manipulation Can be a Breeze
Zhenjia Xu, Cheng Chi, Benjamin Burchfiel +3
This paper introduces DextAIRity, an approach to manipulate deformable objects using active airflow. In contrast to conventional contact-based quasi-static manipulations, DextAIRit…
Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects
Cheng Chi, Benjamin Burchfiel, Eric Cousineau +2
This paper tackles the task of goal-conditioned dynamic manipulation of deformable objects. This task is highly challenging due to its complex dynamics (introduced by object deform…
Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower
Jimmy Wu, Xingyuan Sun, Andy Zeng +3
We investigate pneumatic non-prehensile manipulation (i.e., blowing) as a means of efficiently moving scattered objects into a target receptacle. Due to the chaotic nature of aerod…
TANDEM: Learning Joint Exploration and Decision Making with Tactile Sensors
Jingxi Xu, Shuran Song, Matei Ciocarlie
Inspired by the human ability to perform complex manipulation in the complete absence of vision (like retrieving an object from a pocket), the robotic manipulation field is motivat…
Scene Editing as Teleoperation: A Case Study in 6DoF Kit Assembly
Yulong Li, Shubham Agrawal, Jen-Shuo Liu +2
Studies in robot teleoperation have been centered around action specifications -- from continuous joint control to discrete end-effector pose control. However, these robot-centric…
Learning to See before Learning to Act: Visual Pre-training for Manipulation
Lin Yen-Chen, Andy Zeng, Shuran Song +2
Does having visual priors (e.g. the ability to detect objects) facilitate learning to perform vision-based manipulation (e.g. picking up objects)? We study this problem under the f…
Visual Perspective Taking for Opponent Behavior Modeling
Boyuan Chen, Yuhang Hu, Robert Kwiatkowski +2
In order to engage in complex social interaction, humans learn at a young age to infer what others see and cannot see from a different point-of-view, and learn to predict others' p…
FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding
Huy Ha, Shuran Song
High-velocity dynamic actions (e.g., fling or throw) play a crucial role in our everyday interaction with deformable objects by improving our efficiency and effectively expanding o…
Spatial Intention Maps for Multi-Agent Mobile Manipulation
Jimmy Wu, Xingyuan Sun, Andy Zeng +3
The ability to communicate intention enables decentralized multi-agent robots to collaborate while performing physical tasks. In this work, we present spatial intention maps, a new…