Publications (40)
ChicGrasp: Imitation-Learning based Customized Dual-Jaw Gripper Control for Delicate, Irregular Bio-products Manipulation
Amirreza Davar, Zhengtong Xu, Siavash Mahmoudi +6
Automated poultry processing lines still rely on humans to lift slippery, easily bruised carcasses onto a shackle conveyor. Deformability, anatomical variance, and strict hygiene r…
Imagining the Sense of Touch: Touch-Informed Manipulation via Imagined Tactile Representations
Zhiyuan Zhang, Adeesh Desai, Jyun-Chi Hu +7
Tactile sensing can substantially improve contact-rich robotic manipulation, yet its practical deployment remains limited by the fragility, calibration requirements, and maintenanc…
ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning
Quan Khanh Luu, Pokuang Zhou, Zhengtong Xu +3
Supervised visuomotor policies have shown strong performance in robotic manipulation but often struggle in tasks with limited visual inputs, such as operations in confined spaces a…
In-Hand Singulation and Scooping Manipulation with a 5 DOF Tactile Gripper
Yuhao Zhou, Pokuang Zhou, Shaoxiong Wang +1
Manipulation tasks often require a high degree of dexterity, typically necessitating grippers with multiple degrees of freedom (DoF). While a robotic hand equipped with multiple fi…
Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning
Heng Zhang, Haddy Alchaer, Arash Ajoudani +1
We introduce Reward-Zero, a general-purpose implicit reward mechanism that transforms natural-language task descriptions into dense, semantically grounded progress signals for rein…
Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer
Yunhai Han, Kelin Yu, Rahul Batra +6
Reliable robotic grasping, especially with deformable objects such as fruits, remains a challenging task due to underactuated contact interactions with a gripper, unknown object dy…
TAC-LOCO: Unified Whole-Body Control for Quadrupedal TACtile-Informed LOCO-Manipulation
Muqun Hu, Yuhao Zhou, Kabir Ray Malik +4
The paper introduces TAC-LOCO, a reinforcement learning framework that integrates tactile sensor data with proprioception to enable unified whole-body control of a quadrupedal robo…
Soft Surfaced Vision-Based Tactile Sensing for Bipedal Robot Applications
Jaeeun Kim, Junhee Lim, Yu She
Legged locomotion benefits from embodied sensing, where perception emerges from the physical interaction between body and environment. We present a soft-surfaced, vision-based tact…
LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping
Zhengtong Xu, Yu She
Grasping is a crucial task in robotics, necessitating tactile feedback and reactive grasping adjustments for robust grasping of objects under various conditions and with differing…
Feelit: Combining Compliant Shape Displays with Vision-Based Tactile Sensors for Real-Time Teletaction
Oscar Yu, Yu She
Teletaction, the transmission of tactile feedback or touch, is a crucial aspect in the field of teleoperation. High-quality teletaction feedback allows users to remotely manipulate…
Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing
Yu She, Sandra Q. Liu, Peiyu Yu +1
Soft robots offer significant advantages in adaptability, safety, and dexterity compared to conventional rigid-body robots. However, it is challenging to equip soft robots with acc…
DREAM-Chunk: Reactive Action Chunking with Latent World Model
Wenxi Chen, Kaidi Zhang, Chi Lin +6
Action chunking has become a common interface for vision-language-action (VLA) models, enabling low-frequency policy inference to drive high-frequency robot execution. However, onc…
Caterpillar-Inspired Spring-Based Compressive Continuum Robot for Bristle-based Exploration
Zhixian Hu, Yu She, Juan Wachs
Exploration of confined spaces, such as pipelines and ducts, remains challenging for conventional rigid robots due to limited space, irregular geometry, and restricted access. Insp…
CONTACT: CONtact-aware TACTile Learning for Robotic Disassembly
Yosuke Saka, Jyun-Chi Hu, Adeesh Desai +6
Robotic disassembly involves contact-rich interactions in which successful manipulation depends not only on geometric alignment but also on force-dependent state transitions. While…
A Magnetic-Actuated Vision-Based Whisker Array for Contact Perception and Grasping
Zhixian Hu, Juan Wachs, Yu She
Tactile sensing and the manipulation of delicate objects are critical challenges in robotics. This study presents a vision-based magnetic-actuated whisker array sensor that integra…
TacVLA: Contact-Aware Tactile Fusion for Robust Vision-Language-Action Manipulation
Kaidi Zhang, Heng Zhang, Zhengtong Xu +7
Vision-Language-Action (VLA) models have demonstrated significant advantages in robotic manipulation. However, their reliance on vision and language often leads to suboptimal perfo…
EquiBim: Learning Symmetry-Equivariant Policy for Bimanual Manipulation
Zhiyuan Zhang, Aditya Mohan, Seungho Han +3
Robotic imitation learning has achieved impressive success in learning complex manipulation behaviors from demonstrations. However, many existing robot learning methods do not expl…
EquiForm: Noise-Robust SE(3)-Equivariant Policy Learning from 3D Point Clouds
Zhiyuan Zhang, Yu She
Visual imitation learning with 3D point clouds has advanced robotic manipulation by providing geometry-aware, appearance-invariant observations. However, point cloud-based policies…
GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger
Shaoxiong Wang, Yu She, Branden Romero +1
Vision-based tactile sensors have the potential to provide important contact geometry to localize the objective with visual occlusion. However, it is challenging to measure high-re…
Learning Visual Feature-Based World Models via Residual Latent Action
Xinyu Zhang, Zhengtong Xu, Yutian Tao +3
World models predict future transitions from observations and actions. Existing works predominantly focus on image generation only. Visual feature-based world models, on the other…
Cable Manipulation with a Tactile-Reactive Gripper
Yu She, Shaoxiong Wang, Siyuan Dong +3
Cables are complex, high dimensional, and dynamic objects. Standard approaches to manipulate them often rely on conservative strategies that involve long series of very slow and in…
VILP: Imitation Learning with Latent Video Planning
Zhengtong Xu, Qiang Qiu, Yu She
In the era of generative AI, integrating video generation models into robotics opens new possibilities for the general-purpose robot agent. This paper introduces imitation learning…
SeeBelow: Sub-dermal 3D Reconstruction of Tumors with Surgical Robotic Palpation and Tactile Exploration
Raghava Uppuluri, Abhinaba Bhattacharjee, Sohel Anwar +1
Surgical scene understanding in Robot-assisted Minimally Invasive Surgery (RMIS) is highly reliant on visual cues and lacks tactile perception. Force-modulated surgical palpation w…
Learning When to See and When to Feel: Adaptive Vision-Torque Fusion for Contact-Aware Manipulation
Jiuzhou Lei, Chang Liu, Yu She +2
Vision-based policies have achieved a good performance in robotic manipulation due to the accessibility and richness of visual observations. However, purely visual sensing becomes…
Synthetic Data Augmentation for Enhanced Chicken Carcass Instance Segmentation
Yihong Feng, Chaitanya Pallerla, Xiaomin Lin +5
The poultry industry has been driven by broiler chicken production and has grown into the world's largest animal protein sector. Automated detection of chicken carcasses on process…
DiffOG: Differentiable Policy Trajectory Optimization with Generalizability
Zhengtong Xu, Zichen Miao, Qiang Qiu +2
Imitation learning-based visuomotor policies excel at manipulation tasks but often produce suboptimal action trajectories compared to model-based methods. Directly mapping camera d…
UniT: Data Efficient Tactile Representation with Generalization to Unseen Objects
Zhengtong Xu, Raghava Uppuluri, Xinwei Zhang +5
UniT is an approach to tactile representation learning, using VQGAN to learn a compact latent space and serve as the tactile representation. It uses tactile images obtained from a…
Visuotactile Affordances for Cloth Manipulation with Local Control
Neha Sunil, Shaoxiong Wang, Yu She +2
Cloth in the real world is often crumpled, self-occluded, or folded in on itself such that key regions, such as corners, are not directly graspable, making manipulation difficult.…
PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation
Pengyuan Guo, Zhonghao Mai, Zhengtong Xu +8
Recent advances in vision-language models (VLMs) have enabled increasing progress in real-world robot manipulation. However, long-horizon manipulation in unstructured environments…
Canonical Policy: Learning Canonical 3D Representation for SE(3)-Equivariant Policy
Zhiyuan Zhang, Zhengtong Xu, Jai Nanda Lakamsani +1
Visual Imitation learning has achieved remarkable progress in robotic manipulation, yet generalization to unseen objects, scene layouts, and camera viewpoints remains a key challen…
CompliantVLA-adaptor: VLM-Guided Variable Impedance Action for Safe Contact-Rich Manipulation
Heng Zhang, Wei-Hsing Huang, Qiyi Tong +7
We propose a CompliantVLA-adaptor that augments the state-of-the-art Vision-Language-Action (VLA) models with vision-language model (VLM)-informed context-aware variable impedance…
Safe Learning for Contact-Rich Robot Tasks: A Survey from Classical Learning-Based Methods to Safe Foundation Models
Heng Zhang, Rui Dai, Gokhan Solak +3
Contact-rich tasks pose significant challenges for robotic systems due to inherent uncertainty, complex dynamics, and the high risk of damage during interaction. Recent advances in…
LeTO: Learning Constrained Visuomotor Policy with Differentiable Trajectory Optimization
Zhengtong Xu, Yu She
This paper introduces LeTO, a method for learning constrained visuomotor policy with differentiable trajectory optimization. Our approach integrates a differentiable optimization l…
ContactWorld: What Matters in Vision-Tactile World Models for Contact-Rich Manipulation
Zhiyuan Zhang, Pokuang Zhou, Kaidi Zhang +6
Contact-rich manipulation requires world models to reason over complex contact dynamics from multimodal sensory observations. However, it remains unclear which representation prope…
From Simulation to the Real-World: An In-Field 6D Pose Dataset and Baseline for Robotic Strawberry Harvesting
Woojung Son, Won Suk Lee, Zijing Huang +4
Robotic strawberry harvesting requires precise 6D pose estimation; however, collecting 6D pose ground truth in real agricultural fields is inherently challenging. Existing strawber…
Learning Tactile-Aware Quadrupedal Loco-Manipulation Policies
Pokuang Zhou, Yuhao Zhou, Quan Khanh Luu +7
Quadrupedal loco-manipulation is commonly built on visual perception and proprioception. Yet reliable contact-rich manipulation remains difficult: vision and proprioception alone c…
MuxGel: Simultaneous Dual-Modal Visuo-Tactile Sensing via Spatially Multiplexing and Deep Reconstruction
Zhixian Hu, Zhengtong Xu, Sheeraz Athar +2
High-fidelity visuo-tactile sensing is important for precise robotic manipulation, yet most vision-based tactile sensors rely on opaque coatings that enable tactile sensing but blo…
VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation
Yuhao Zhou, Sheeraz Athar, Zhixian Hu +4
This paper presents a tactile-reactive gripper that integrates a Visuo-Tactile Active Palm (VTAP) and compliant, reconfigurable fingers equipped with tactile array sensors. The des…
MVP-Tac: A Miniaturized Dual-Modal Vision and Photoelastic Tactile Sensor for Robot-Assisted Minimally Invasive Surgery
Md Rakibul Islam Prince, Jaeeun Kim, Yuhao Zhou +9
Robot-assisted minimally invasive surgery (RMIS) offers major benefits over open and conventional laparoscopic procedures, yet it still lacks tactile feedback for palpation while o…
Robot Tape Manipulation for 3D Printing
Nahid Tushar, Rencheng Wu, Yu She +2
3D printing has enabled various applications using different forms of materials, such as filaments, sheets, and inks. Typically, during 3D printing, feedstocks are transformed into…