312 citations · 2.2k across the 74 of their papers we have counts for
31 papers · 1 filter
MUTEX: Learning Unified Policies from Multimodal Task Specifications
Rutav Shah, Roberto Martín-Martín, Yuke Zhu
Humans use different modalities, such as speech, text, images, videos, etc., to communicate their intent and goals with teammates. For robots to become better assistants, we aim to…
Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation
Mingyo Seo, Steve Han, Kyutae Sim +4
We tackle the problem of developing humanoid loco-manipulation skills with deep imitation learning. The difficulty of collecting task demonstrations and training policies for human…
Symbolic State Space Optimization for Long Horizon Mobile Manipulation Planning
Xiaohan Zhang, Yifeng Zhu, Yan Ding +4
In existing task and motion planning (TAMP) research, it is a common assumption that experts manually specify the state space for task-level planning. A well-developed state space…
Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception
Cheng-Chun Hsu, Zhenyu Jiang, Yuke Zhu
Virtualizing the physical world into virtual models has been a critical technique for robot navigation and planning in the real world. To foster manipulation with articulated objec…
MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
Chen Wang, Linxi Fan, Jiankai Sun +5
Imitation learning from human demonstrations is a promising paradigm for teaching robots manipulation skills in the real world. However, learning complex long-horizon tasks often r…
Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment
Huihan Liu, Soroush Nasiriany, Lance Zhang +2
With the rapid growth of computing powers and recent advances in deep learning, we have witnessed impressive demonstrations of novel robot capabilities in research settings. Noneth…
VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors
Yifeng Zhu, Abhishek Joshi, Peter Stone +1
We introduce VIOLA, an object-centric imitation learning approach to learning closed-loop visuomotor policies for robot manipulation. Our approach constructs object-centric represe…
VIMA: General Robot Manipulation with Multimodal Prompts
Yunfan Jiang, Agrim Gupta, Zichen Zhang +7
Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task speci…
Learning to Walk by Steering: Perceptive Quadrupedal Locomotion in Dynamic Environments
Mingyo Seo, Ryan Gupta, Yifeng Zhu +3
We tackle the problem of perceptive locomotion in dynamic environments. In this problem, a quadrupedal robot must exhibit robust and agile walking behaviors in response to environm…
Visually Grounded Task and Motion Planning for Mobile Manipulation
Xiaohan Zhang, Yifeng Zhu, Yan Ding +3
Task and motion planning (TAMP) algorithms aim to help robots achieve task-level goals, while maintaining motion-level feasibility. This paper focuses on TAMP domains that involve…
OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation
Josiah Wong, Viktor Makoviychuk, Anima Anandkumar +1
Learning performant robot manipulation policies can be challenging due to high-dimensional continuous actions and complex physics-based dynamics. This can be alleviated through int…
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
Ajay Mandlekar, Danfei Xu, Josiah Wong +7
Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch…
Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation
Yifeng Zhu, Peter Stone, Yuke Zhu
We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demons…
Discovering Generalizable Skills via Automated Generation of Diverse Tasks
Kuan Fang, Yuke Zhu, Silvio Savarese +1
The learning efficiency and generalization ability of an intelligent agent can be greatly improved by utilizing a useful set of skills. However, the design of robot skills can ofte…
Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations
Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik +2
Grasp detection in clutter requires the robot to reason about the 3D scene from incomplete and noisy perception. In this work, we draw insight that 3D reconstruction and grasp lear…
Emergent Hand Morphology and Control from Optimizing Robust Grasps of Diverse Objects
Xinlei Pan, Animesh Garg, Animashree Anandkumar +1
Evolution in nature illustrates that the creatures' biological structure and their sensorimotor skills adapt to the environmental changes for survival. Likewise, the ability to mor…
Learning Multi-Arm Manipulation Through Collaborative Teleoperation
Albert Tung, Josiah Wong, Ajay Mandlekar +4
Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but ha…
Human-in-the-Loop Imitation Learning using Remote Teleoperation
Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín +3
Imitation Learning is a promising paradigm for learning complex robot manipulation skills by reproducing behavior from human demonstrations. However, manipulation tasks often conta…
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs
Yifeng Zhu, Jonathan Tremblay, Stan Birchfield +1
We present a visually grounded hierarchical planning algorithm for long-horizon manipulation tasks. Our algorithm offers a joint framework of neuro-symbolic task planning and low-l…
Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
Michelle A. Lee, Matthew Tan, Yuke Zhu +1
Using sensor data from multiple modalities presents an opportunity to encode redundant and complementary features that can be useful when one modality is corrupted or noisy. Humans…
Deep Affordance Foresight: Planning Through What Can Be Done in the Future
Danfei Xu, Ajay Mandlekar, Roberto Martín-Martín +3
Planning in realistic environments requires searching in large planning spaces. Affordances are a powerful concept to simplify this search, because they model what actions can be s…
Fast Uncertainty Quantification for Deep Object Pose Estimation
Guanya Shi, Yifeng Zhu, Jonathan Tremblay +4
Deep learning-based object pose estimators are often unreliable and overconfident especially when the input image is outside the training domain, for instance, with sim2real transf…
Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion
Xingye Da, Zhaoming Xie, David Hoeller +5
We present a hierarchical framework that combines model-based control and reinforcement learning (RL) to synthesize robust controllers for a quadruped (the Unitree Laikago). The sy…
KETO: Learning Keypoint Representations for Tool Manipulation
Zengyi Qin, Kuan Fang, Yuke Zhu +2
We aim to develop an algorithm for robots to manipulate novel objects as tools for completing different task goals. An efficient and informative representation would facilitate the…