25 citations · 56 across the 13 of their papers we have counts for
14 papers
Lightning Grasp: High Performance Procedural Grasp Synthesis with Contact Fields
Zhao-Heng Yin, Pieter Abbeel
Despite years of research, real-time diverse grasp synthesis for dexterous hands remains an unsolved core challenge in robotics and computer graphics. We present Lightning Grasp, a…
Object-centric 3D Motion Field for Robot Learning from Human Videos
Zhao-Heng Yin, Sherry Yang, Pieter Abbeel
Learning robot control policies from human videos is a promising direction for scaling up robot learning. However, how to extract action knowledge (or action representations) from…
RoboCopilot: Human-in-the-loop Interactive Imitation Learning for Robot Manipulation
Philipp Wu, Yide Shentu, Qiayuan Liao +5
Learning from human demonstration is an effective approach for learning complex manipulation skills. However, existing approaches heavily focus on learning from passive human demon…
DexterityGen: Foundation Controller for Unprecedented Dexterity
Zhao-Heng Yin, Changhao Wang, Luis Pineda +11
Teaching robots dexterous manipulation skills, such as tool use, presents a significant challenge. Current approaches can be broadly categorized into two strategies: human teleoper…
Hand-Object Interaction Pretraining from Videos
Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sferrazza +4
We present an approach to learn general robot manipulation priors from 3D hand-object interaction trajectories. We build a framework to use in-the-wild videos to generate sensorimo…
Semi-Supervised One-Shot Imitation Learning
Philipp Wu, Kourosh Hakhamaneshi, Yuqing Du +3
One-shot Imitation Learning~(OSIL) aims to imbue AI agents with the ability to learn a new task from a single demonstration. To supervise the learning, OSIL typically requires a pr…
From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control
Yide Shentu, Philipp Wu, Aravind Rajeswaran +1
Hierarchical control for robotics has long been plagued by the need to have a well defined interface layer to communicate between high-level task planners and low-level policies. W…
Twisting Lids Off with Two Hands
Toru Lin, Zhao-Heng Yin, Haozhi Qi +2
Manipulating objects with two multi-fingered hands has been a long-standing challenge in robotics, due to the contact-rich nature of many manipulation tasks and the complexity inhe…
Video as the New Language for Real-World Decision Making
Sherry Yang, Jacob Walker, Jack Parker-Holder +5
Both text and video data are abundant on the internet and support large-scale self-supervised learning through next token or frame prediction. However, they have not been equally l…
Scalable Diffusion for Materials Generation
Sherry Yang, KwangHwan Cho, Amil Merchant +4
Generative models trained on internet-scale data are capable of generating novel and realistic texts, images, and videos. A natural next question is whether these models can advanc…
Video Language Planning
Yilun Du, Mengjiao Yang, Pete Florence +10
We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pr…
Interactive Task Planning with Language Models
Boyi Li, Philipp Wu, Pieter Abbeel +1
An interactive robot framework accomplishes long-horizon task planning and can easily generalize to new goals and distinct tasks, even during execution. However, most traditional m…
Learning Interactive Real-World Simulators
Sherry Yang, Yilun Du, Kamyar Ghasemipour +4
Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative models is to simulate re…
RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning
Kevin Zakka, Philipp Wu, Laura Smith +8
Replicating human-like dexterity in robot hands represents one of the largest open problems in robotics. Reinforcement learning is a promising approach that has achieved impressive…