68 citations · 139 across the 18 of their papers we have counts for
18 papers
Best of Sim and Real: Decoupled Visuomotor Manipulation via Learning Control in Simulation and Perception in Real
Jialei Huang, Zhaoheng Yin, Yingdong Hu +3
Sim-to-real transfer remains a fundamental challenge in robot manipulation due to the entanglement of perception and control in end-to-end learning. We present a decoupled framewor…
Learning Generalizable Tool-use Skills through Trajectory Generation
Carl Qi, Yilin Wu, Lifan Yu +4
Autonomous systems that efficiently utilize tools can assist humans in completing many common tasks such as cooking and cleaning. However, current systems fall short of matching hu…
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators
Philipp Wu, Yide Shentu, Zhongke Yi +2
Humans can teleoperate robots to accomplish complex manipulation tasks. Imitation learning has emerged as a powerful framework that leverages human teleoperated demonstrations to t…
SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks
Xingyu Lin, John So, Sashwat Mahalingam +2
The existing internet-scale image and video datasets cover a wide range of everyday objects and tasks, bringing the potential of learning policies that generalize in diverse scenar…
RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects
Zhenjia Xu, Zhou Xian, Xingyu Lin +4
We introduce RoboNinja, a learning-based cutting system for multi-material objects (i.e., soft objects with rigid cores such as avocados or mangos). In contrast to prior works usin…
Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking
Zixuan Huang, Xingyu Lin, David Held
State estimation is one of the greatest challenges for cloth manipulation due to cloth's high dimensionality and self-occlusion. Prior works propose to identify the full state of c…
Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation
Xingyu Lin, Carl Qi, Yunchu Zhang +5
Effective planning of long-horizon deformable object manipulation requires suitable abstractions at both the spatial and temporal levels. Previous methods typically either focus on…
Learning Closed-loop Dough Manipulation Using a Differentiable Reset Module
Carl Qi, Xingyu Lin, David Held
Deformable object manipulation has many applications such as cooking and laundry folding in our daily lives. Manipulating elastoplastic objects such as dough is particularly challe…
Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation
Zixuan Huang, Xingyu Lin, David Held
Self-occlusion is challenging for cloth manipulation, as it makes it difficult to estimate the full state of the cloth. Ideally, a robot trying to unfold a crumpled or folded cloth…
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools
Xingyu Lin, Zhiao Huang, Yunzhu Li +3
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators provide gradients to…
Self-supervised Transparent Liquid Segmentation for Robotic Pouring
Gautham Narayan Narasimhan, Kai Zhang, Ben Eisner +2
Liquid state estimation is important for robotics tasks such as pouring; however, estimating the state of transparent liquids is a challenging problem. We propose a novel segmentat…
Learning Visible Connectivity Dynamics for Cloth Smoothing
Xingyu Lin, Yufei Wang, Zixuan Huang +1
Robotic manipulation of cloth remains challenging for robotics due to the complex dynamics of the cloth, lack of a low-dimensional state representation, and self-occlusions. In con…
ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoning
Yufei Wang, Gautham Narayan Narasimhan, Xingyu Lin +2
Current image-based reinforcement learning (RL) algorithms typically operate on the whole image without performing object-level reasoning. This leads to inefficient goal sampling a…
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
Xingyu Lin, Yufei Wang, Jake Olkin +1
Manipulating deformable objects has long been a challenge in robotics due to its high dimensional state representation and complex dynamics. Recent success in deep reinforcement le…
Reinforcement Learning without Ground-Truth State
Xingyu Lin, Harjatin Singh Baweja, David Held
To perform robot manipulation tasks, a low-dimensional state of the environment typically needs to be estimated. However, designing a state estimator can sometimes be difficult, es…
Adaptive Variance for Changing Sparse-Reward Environments
Xingyu Lin, Pengsheng Guo, Carlos Florensa +1
Robots that are trained to perform a task in a fixed environment often fail when facing unexpected changes to the environment due to a lack of exploration. We propose a principled…
Learning Robust Object Recognition Using Composed Scenes from Generative Models
Hao Wang, Xingyu Lin, Yimeng Zhang +1
Recurrent feedback connections in the mammalian visual system have been hypothesized to play a role in synthesizing input in the theoretical framework of analysis by synthesis. The…
Transfer of View-manifold Learning to Similarity Perception of Novel Objects
Xingyu Lin, Hao Wang, Zhihao Li +3
We develop a model of perceptual similarity judgment based on re-training a deep convolution neural network (DCNN) that learns to associate different views of each 3D object to cap…