553 citations · 783 across the 28 of their papers we have counts for
28 papers
Coupled Distributional Random Expert Distillation for World Model Online Imitation Learning
Shangzhe Li, Zhiao Huang, Hao Su
Imitation Learning (IL) has achieved remarkable success across various domains, including robotics, autonomous driving, and healthcare, by enabling agents to learn complex behavior…
Diffusion Dynamics Models with Generative State Estimation for Cloth Manipulation
Tongxuan Tian, Haoyang Li, Bo Ai +3
Cloth manipulation is challenging due to its highly complex dynamics, near-infinite degrees of freedom, and frequent self-occlusions, which complicate both state estimation and dyn…
Reward-free World Models for Online Imitation Learning
Shangzhe Li, Zhiao Huang, Hao Su
Imitation learning (IL) enables agents to acquire skills directly from expert demonstrations, providing a compelling alternative to reinforcement learning. However, prior online IL…
ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI
Stone Tao, Fanbo Xiang, Arth Shukla +20
Simulation has enabled unprecedented compute-scalable approaches to robot learning. However, many existing simulation frameworks typically support a narrow range of scenes/tasks an…
DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
Zhiao Huang, Feng Chen, Yewen Pu +3
Combining gradient-based trajectory optimization with differentiable physics simulation is an efficient technique for solving soft-body manipulation problems. Using a well-crafted…
Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Dataset
Litian Liang, Liuyu Bian, Caiwei Xiao +6
Building robots that can automate labor-intensive tasks has long been the core motivation behind the advancements in computer vision and the robotics community. Recent interest in…
Reparameterized Policy Learning for Multimodal Trajectory Optimization
Zhiao Huang, Litian Liang, Zhan Ling +3
We investigate the challenge of parametrizing policies for reinforcement learning (RL) in high-dimensional continuous action spaces. Our objective is to develop a multimodal policy…
Deductive Verification of Chain-of-Thought Reasoning
Zhan Ling, Yunhao Fang, Xuanlin Li +4
Large Language Models (LLMs) significantly benefit from Chain-of-Thought (CoT) prompting in performing various reasoning tasks. While CoT allows models to produce more comprehensiv…
MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field
Kaizhi Yang, Xiaoshuai Zhang, Zhiao Huang +3
We present MovingParts, a NeRF-based method for dynamic scene reconstruction and part discovery. We consider motion as an important cue for identifying parts, that all particles on…
Chain-of-Thought Predictive Control
Zhiwei Jia, Vineet Thumuluri, Fangchen Liu +3
We study generalizable policy learning from demonstrations for complex low-level control (e.g., contact-rich object manipulations). We propose a novel hierarchical imitation learni…
DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics
Sizhe Li, Zhiao Huang, Tao Chen +4
In this work, we aim to learn dexterous manipulation of deformable objects using multi-fingered hands. Reinforcement learning approaches for dexterous rigid object manipulation wou…
RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects
Zhenjia Xu, Zhou Xian, Xingyu Lin +4
We introduce RoboNinja, a learning-based cutting system for multi-material objects (i.e., soft objects with rigid cores such as avocados or mangos). In contrast to prior works usin…
ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
Jiayuan Gu, Fanbo Xiang, Xuanlin Li +12
Generalizable manipulation skills, which can be composed to tackle long-horizon and complex daily chores, are one of the cornerstones of Embodied AI. However, existing benchmarks,…
Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Xingyu Lin, Carl Qi, Yunchu Zhang +5
Effective planning of long-horizon deformable object manipulation requires suitable abstractions at both the spatial and temporal levels. Previous methods typically either focus on…
Abstract-to-Executable Trajectory Translation for One-Shot Task Generalization
Stone Tao, Xiaochen Li, Tongzhou Mu +3
Training long-horizon robotic policies in complex physical environments is essential for many applications, such as robotic manipulation. However, learning a policy that can genera…
RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks
Haochen Shi, Huazhe Xu, Zhiao Huang +2
Modeling and manipulating elasto-plastic objects are essential capabilities for robots to perform complex industrial and household interaction tasks (e.g., stuffing dumplings, roll…
Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics
Sizhe Li, Zhiao Huang, Tao Du +3
Differentiable physics has recently been shown as a powerful tool for solving soft-body manipulation tasks. However, the differentiable physics solver often gets stuck when the ini…
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
Xingyu Lin, Zhiao Huang, Yunzhu Li +3
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators provide gradients to…
Learning Multi-Object Dynamics with Compositional Neural Radiance Fields
Danny Driess, Zhiao Huang, Yunzhu Li +2
We present a method to learn compositional multi-object dynamics models from image observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and graph neural…
Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation
Xiaoshuai Zhang, Rui Chen, Ang Li +11
In this paper, we focus on the simulation of active stereovision depth sensors, which are popular in both academic and industry communities. Inspired by the underlying mechanism of…
ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Tongzhou Mu, Zhan Ling, Fanbo Xiang +6
Object manipulation from 3D visual inputs poses many challenges on building generalizable perception and policy models. However, 3D assets in existing benchmarks mostly lack the di…
PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics
Zhiao Huang, Yuanming Hu, Tao Du +4
Simulated virtual environments serve as one of the main driving forces behind developing and evaluating skill learning algorithms. However, existing environments typically only sim…
Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous Graph Neural Networks
Hao Tang, Zhiao Huang, Jiayuan Gu +2
Current graph neural networks (GNNs) lack generalizability with respect to scales (graph sizes, graph diameters, edge weights, etc..) when solving many graph analysis problems. Tak…
Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories
Tiange Luo, Kaichun Mo, Zhiao Huang +4
We address the problem of discovering 3D parts for objects in unseen categories. Being able to learn the geometry prior of parts and transfer this prior to unseen categories pose f…