Publications (60)
Limits of dispersoid size and number density in oxide dispersion strengthened alloys fabricated with powder bed fusion-laser beam
Nathan A. Wassermann, Yongchang Li, Alexander J. Myers +7
Previous work on additively-manufactured oxide dispersion strengthened alloys focused on experimental approaches, resulting in larger dispersoid sizes and lower number densities th…
RoboAssembly: Learning Generalizable Furniture Assembly Policy in a Novel Multi-robot Contact-rich Simulation Environment
Mingxin Yu, Lin Shao, Zhehuan Chen +4
Part assembly is a typical but challenging task in robotics, where robots assemble a set of individual parts into a complete shape. In this paper, we develop a robotic assembly sim…
ClusterNet: 3D Instance Segmentation in RGB-D Images
Lin Shao, Ye Tian, Jeannette Bohg
We propose a method for instance-level segmentation that uses RGB-D data as input and provides detailed information about the location, geometry and number of individual objects in…
Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA
Tutian Tang, Xingyu Ji, Wanli Xing +7
While Vision-Language-Action (VLA) models have demonstrated remarkable success in robotic manipulation, their application has largely been confined to low-degree-of-freedom end-eff…
ManiFoundation Model for General-Purpose Robotic Manipulation of Contact Synthesis with Arbitrary Objects and Robots
Zhixuan Xu, Chongkai Gao, Zixuan Liu +10
To substantially enhance robot intelligence, there is a pressing need to develop a large model that enables general-purpose robots to proficiently undertake a broad spectrum of man…
Learning to Regrasp by Learning to Place
Shuo Cheng, Kaichun Mo, Lin Shao
In this paper, we explore whether a robot can learn to regrasp a diverse set of objects to achieve various desired grasp poses. Regrasping is needed whenever a robot's current gras…
Goal-VLA: Image-Generative VLMs as Object-Centric World Models Empowering Zero-shot Robot Manipulation
Haonan Chen, Jingxiang Guo, Bangjun Wang +7
Generalization remains a fundamental challenge in robotic manipulation. To tackle this challenge, recent Vision-Language-Action (VLA) models build policies on top of Vision-Languag…
Design and Control of Roller Grasper V2 for In-Hand Manipulation
Shenli Yuan, Lin Shao, Connor L. Yako +2
The ability to perform in-hand manipulation still remains an unsolved problem; having this capability would allow robots to perform sophisticated tasks requiring repositioning and…
Learning to Scaffold the Development of Robotic Manipulation Skills
Lin Shao, Toki Migimatsu, Jeannette Bohg
Learning contact-rich, robotic manipulation skills is a challenging problem due to the high-dimensionality of the state and action space as well as uncertainty from noisy sensors a…
SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering
Jun Lv, Yunhai Feng, Cheng Zhang +3
Model-based reinforcement learning (MBRL) is recognized with the potential to be significantly more sample-efficient than model-free RL. How an accurate model can be developed auto…
UniGrasp: Learning a Unified Model to Grasp with Multifingered Robotic Hands
Lin Shao, Fabio Ferreira, Mikael Jorda +6
To achieve a successful grasp, gripper attributes such as its geometry and kinematics play a role as important as the object geometry. The majority of previous work has focused on…
DexSinGrasp: Learning a Unified Policy for Dexterous Object Singulation and Grasping in Densely Cluttered Environments
Lixin Xu, Zixuan Liu, Zhewei Gui +6
Grasping objects in cluttered environments remains a fundamental yet challenging problem in robotic manipulation. While prior works have explored learning-based synergies between p…
Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55
Li Yi, Lin Shao, Manolis Savva +47
We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database. The benchmark consists of two tasks: part-level segmentation…
Differentiable Contact Dynamics for Stable Object Placement Under Geometric Uncertainties
Linfeng Li, Gang Yang, Lin Shao +1
From serving a cup of coffee to positioning mechanical parts during assembly, stable object placement is a crucial skill for future robots. It becomes particularly challenging unde…
ManiLong-Shot: Interaction-Aware One-Shot Imitation Learning for Long-Horizon Manipulation
Zixuan Chen, Chongkai Gao, Lin Shao +3
One-shot imitation learning (OSIL) offers a promising way to teach robots new skills without large-scale data collection. However, current OSIL methods are primarily limited to sho…
Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features
Chengkai Hou, Zhengrong Xue, Bingyang Zhou +3
Detecting 3D keypoints with semantic consistency is widely used in many scenarios such as pose estimation, shape registration and robotics. Currently, most unsupervised 3D keypoint…
Generative 3D Part Assembly via Dynamic Graph Learning
Jialei Huang, Guanqi Zhan, Qingnan Fan +5
Autonomous part assembly is a challenging yet crucial task in 3D computer vision and robotics. Analogous to buying an IKEA furniture, given a set of 3D parts that can assemble a si…
GRAC: Self-Guided and Self-Regularized Actor-Critic
Lin Shao, Yifan You, Mengyuan Yan +2
Deep reinforcement learning (DRL) algorithms have successfully been demonstrated on a range of challenging decision making and control tasks. One dominant component of recent deep…
ClothesNet: An Information-Rich 3D Garment Model Repository with Simulated Clothes Environment
Bingyang Zhou, Haoyu Zhou, Tianhai Liang +10
We present ClothesNet: a large-scale dataset of 3D clothes objects with information-rich annotations. Our dataset consists of around 4400 models covering 11 categories annotated wi…
ContactExplorer: Contact Coverage-Guided Exploration for General-Purpose Dexterous Manipulation
Zixuan Liu, Ruoyi Qiao, Chenrui Tie +5
Reinforcement learning has achieved remarkable success in domains such as Atari games, navigation, and locomotion, where exploration can often be guided by novelty over states or d…
Grasp: A Unified Representation of Robot and Object Interaction for Cross-Embodiment Dexterous Grasping
Zhenyu Wei, Zhixuan Xu, Jingxiang Guo +5
Dexterous grasping is a fundamental yet challenging skill in robotic manipulation, requiring precise interaction between robotic hands and objects. In this paper, we present $\math…
VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models
Chongkai Gao, Zixuan Liu, Zhenghao Chi +8
Recent studies on Vision-Language-Action (VLA) models have shifted from the end-to-end action-generation paradigm toward a pipeline involving task planning followed by action gener…
Dexterous Manipulation Primitives for the Real Robot Challenge
Claire Chen, Krishnan Srinivasan, Jeffrey Zhang +7
This report describes our approach for Phase 3 of the Real Robot Challenge. To solve cuboid manipulation tasks of varying difficulty, we decompose each task into the following prim…
TelePreview: A User-Friendly Teleoperation System with Virtual Arm Assistance for Enhanced Effectiveness
Jingxiang Guo, Jiayu Luo, Zhenyu Wei +5
Teleoperation provides an effective way to collect robot data, which is crucial for learning from demonstrations. In this field, teleoperation faces several key challenges: user-fr…
Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models
Chenrui Tie, Shengxiang Sun, Yudi Lin +9
Assembly hinges on reliably forming connections between parts; yet most robotic approaches plan assembly sequences and part poses while treating connectors as an afterthought. Conn…
LiDARDraft: Generating LiDAR Point Cloud from Versatile Inputs
Haiyun Wei, Fan Lu, Yunwei Zhu +7
Generating realistic and diverse LiDAR point clouds is crucial for autonomous driving simulation. Although previous methods achieve LiDAR point cloud generation from user inputs, t…
Active Contact Sensing for Robust Robot-to-Human Object Handover
Linfeng Li, Lin Shao, David Hsu
Robot-to-human object handover is an essential skill for robot assistants, from serving drinks at home to passing surgical tools in the operating room. We expect robots to perform…
Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models
Chenrui Tie, Shengxiang Sun, Jinxuan Zhu +7
Humans possess an extraordinary ability to understand and execute complex manipulation tasks by interpreting abstract instruction manuals. For robots, however, this capability rema…
DexRepNet++: Learning Dexterous Robotic Manipulation with Geometric and Spatial Hand-Object Representations
Qingtao Liu, Zhengnan Sun, Yu Cui +5
Robotic dexterous manipulation is a challenging problem due to high degrees of freedom (DoFs) and complex contacts of multi-fingered robotic hands. Many existing deep reinforcement…
Adaptive Wall-Following Control for Unmanned Ground Vehicles Using Spiking Neural Networks
Hengye Yang, Yanxiao Chen, Zexuan Fan +2
Unmanned ground vehicles operating in complex environments must adaptively adjust to modeling uncertainties and external disturbances to perform tasks such as wall following and ob…
Learning Reward for Robot Skills Using Large Language Models via Self-Alignment
Yuwei Zeng, Yao Mu, Lin Shao
Learning reward functions remains the bottleneck to equip a robot with a broad repertoire of skills. Large Language Models (LLM) contain valuable task-related knowledge that can po…
DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations
Qingtao Liu, Yu Cui, Qi Ye +5
Robotic dexterous grasping is a challenging problem due to the high degree of freedom (DoF) and complex contacts of multi-fingered robotic hands. Existing deep reinforcement learni…
FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Model
Chongkai Gao, Haozhuo Zhang, Zhixuan Xu +2
We aim to develop a model-based planning framework for world models that can be scaled with increasing model and data budgets for general-purpose manipulation tasks with only langu…
Diff-Transfer: Model-based Robotic Manipulation Skill Transfer via Differentiable Physics Simulation
Yuqi Xiang, Feitong Chen, Qinsi Wang +5
The capability to transfer mastered skills to accomplish a range of similar yet novel tasks is crucial for intelligent robots. In this work, we introduce ,…
Generalizable Long-Horizon Manipulations with Large Language Models
Haoyu Zhou, Mingyu Ding, Weikun Peng +3
This work introduces a framework harnessing the capabilities of Large Language Models (LLMs) to generate primitive task conditions for generalizable long-horizon manipulations with…
Jade: A Differentiable Physics Engine for Articulated Rigid Bodies with Intersection-Free Frictional Contact
Gang Yang, Siyuan Luo, Lin Shao
We present Jade, a differentiable physics engine for articulated rigid bodies. Jade models contacts as the Linear Complementarity Problem (LCP). Compared to existing differentiable…
EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents
Junting Chen, Checheng Yu, Xunzhe Zhou +7
Heterogeneous multi-robot systems (HMRS) have emerged as a powerful approach for tackling complex tasks that single robots cannot manage alone. Current large-language-model-based m…
AdaptPNP: Integrating Prehensile and Non-Prehensile Skills for Adaptive Robotic Manipulation
Jinxuan Zhu, Chenrui Tie, Xinyi Cao +8
Non-prehensile (NP) manipulation, in which robots alter object states without forming stable grasps (for example, pushing, poking, or sliding), significantly broadens robotic manip…
Proton-fluence dependent magnetic properties of exfoliable quasi-2D van der Waals Cr2Si2Te6 magnet
Hector Iturriaga, Ju Chen, Jing Yang +6
The discovery of long-range magnetic ordering in atomically thin materials catapulted the van der Waals (vdW) family of compounds into an unprecedented popularity. In particular, w…
MetaFold: Language-Guided Multi-Category Garment Folding Framework via Trajectory Generation and Foundation Model
Haonan Chen, Junxiao Li, Ruihai Wu +9
Garment folding is a common yet challenging task in robotic manipulation. The deformability of garments leads to a vast state space and complex dynamics, which complicates precise…
SoftMAC: Differentiable Soft Body Simulation with Forecast-based Contact Model and Two-way Coupling with Articulated Rigid Bodies and Clothes
Min Liu, Gang Yang, Siyuan Luo +1
Differentiable physics simulation provides an avenue to tackle previously intractable challenges through gradient-based optimization, thereby greatly improving the efficiency of so…
DSVM-UNet : Enhancing VM-UNet with Dual Self-distillation for Medical Image Segmentation
Renrong Shao, Dongyang Li, Dong Xia +4
Vision Mamba models have been extensively researched in various fields, which address the limitations of previous models by effectively managing long-range dependencies with a line…
OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation
Yifan You, Lin Shao, Toki Migimatsu +1
In this paper, we explore whether a robot can learn to hang arbitrary objects onto a diverse set of supporting items such as racks or hooks. Endowing robots with such an ability ha…
A Visual Approach for Health Information Exploration: Adaptive Levels of Visual Granularity and Interaction Analysis
Stefan Lengauer, Lin Shao, Hossein Miri +12
The effective and targeted provision of health information to consumers, specifically tailored to their needs and preferences, is indispensable in healthcare. With access to approp…
SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning
Jun Lv, Qiaojun Yu, Lin Shao +3
Building general-purpose robots to perform a diverse range of tasks in a large variety of environments in the physical world at the human level is extremely challenging. It require…
FingerEye: Learning Dexterous Manipulation with Continuous Vision-Tactile Sensing
Zhixuan Xu, Yichen Li, Xuanye Wu +2
Dexterous robotic manipulation requires perception that remains informative from pre-contact approach to contact initiation and post-contact control. We introduce FingerEye, a sens…
ShapeForce: Low-Cost Soft Robotic Wrist for Contact-Rich Manipulation
Jinxuan Zhu, Zihao Yan, Yangyu Xiao +5
Contact feedback is essential for contact-rich robotic manipulation, as it allows the robot to detect subtle interaction changes and adjust its actions accordingly. Six-axis force-…
T(R,O) Grasp: Efficient Graph Diffusion of Robot-Object Spatial Transformation for Cross-Embodiment Dexterous Grasping
Xin Fei, Zhixuan Xu, Huaicong Fang +2
Dexterous grasping remains a central challenge in robotics due to the complexity of its high-dimensional state and action space. We introduce T(R,O) Grasp, a diffusion-based framew…
LISN: Language-Instructed Social Navigation with VLM-based Controller Modulating
Junting Chen, Yunchuan Li, Panfeng Jiang +5
Towards human-robot coexistence, socially aware navigation is significant for mobile robots. Yet existing studies on this area focus mainly on path efficiency and pedestrian collis…
Learning Visual Dynamics Models of Rigid Objects using Relational Inductive Biases
Fabio Ferreira, Lin Shao, Tamim Asfour +1
Endowing robots with human-like physical reasoning abilities remains challenging. We argue that existing methods often disregard spatio-temporal relations and by using Graph Neural…
Bi-Adapt: Few-shot Bimanual Adaptation for Novel Categories of 3D Objects via Semantic Correspondence
Jinxian Zhou, Ruihai Wu, Yiwei Liu +5
Bimanual manipulation is imperative yet challenging for robots to execute complex tasks, requiring coordinated collaboration between two arms. However, existing methods for bimanua…
Motion-based Object Segmentation based on Dense RGB-D Scene Flow
Lin Shao, Parth Shah, Vikranth Dwaracherla +1
Given two consecutive RGB-D images, we propose a model that estimates a dense 3D motion field, also known as scene flow. We take advantage of the fact that in robot manipulation sc…
Category-Level Multi-Part Multi-Joint 3D Shape Assembly
Yichen Li, Kaichun Mo, Yueqi Duan +5
Shape assembly composes complex shapes geometries by arranging simple part geometries and has wide applications in autonomous robotic assembly and CAD modeling. Existing works focu…
GAMMA: Generalizable Articulation Modeling and Manipulation for Articulated Objects
Qiaojun Yu, Junbo Wang, Wenhai Liu +5
Articulated objects like cabinets and doors are widespread in daily life. However, directly manipulating 3D articulated objects is challenging because they have diverse geometrical…
OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
Junting Chen, Haotian Liang, Lingxiao Du +8
The rapid progress of navigation, manipulation, and vision models has made mobile manipulators capable in many specialized tasks. However, the open-world mobile manipulation (OWMM)…
RoTri-Diff: A Spatial Robot-Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation
Zixuan Chen, Nga Teng Chan, Yiwen Hou +8
Bimanual manipulation is a fundamental robotic skill that requires continuous and precise coordination between two arms. While imitation learning (IL) is the dominant paradigm for…
RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation
Chongkai Gao, Zhengrong Xue, Shuying Deng +4
We present RiEMann, an end-to-end near Real-time SE(3)-Equivariant Robot Manipulation imitation learning framework from scene point cloud input. Compared to previous methods that r…
TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approach
Weikun Peng, Jun Lv, Yuwei Zeng +5
The tie-knotting task is highly challenging due to the tie's high deformation and long-horizon manipulation actions. This work presents TieBot, a Real-to-Sim-to-Real learning from…
Asymmetric Lattice Disorder Induced at Oxide Interfaces
Steven R. Spurgeon, Tiffany C. Kaspar, Vaithiyalingam Shutthanandan +3
Control of order-disorder phase transitions is a fundamental materials science challenge, underpinning the development of energy storage technologies such as solid oxide fuel cells…
Learning 3D Part Assembly from a Single Image
Yichen Li, Kaichun Mo, Lin Shao +2
Autonomous assembly is a crucial capability for robots in many applications. For this task, several problems such as obstacle avoidance, motion planning, and actuator control have…