233 citations · 451 across the 42 of their papers we have counts for
48 papers
Whole-Body Dexterity
Conor Mc Gartoll, An Dang, Pranay Thangeda +5
Current humanoid dexterous manipulation remains largely confined to end-effectors and simple grasps, falling short of humans' ability to engage the entire body by bracing and slidi…
TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning
Dayou Li, Hao Wang, Qianqian Yang +23
Large-scale egocentric human interaction data is becoming an important source of physical supervision for embodied learning, yet video alone leaves the contact and pressure that ch…
Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience
Raymond Yu, William Huey, Mustafa Mukadam +2
Robots trained on real world data tend to be imprecise, slow, and brittle to perturbations. Improving these policies with reinforcement learning (RL) is an appealing alternative, b…
PTLD: Sim-to-real Privileged Tactile Latent Distillation for Dexterous Manipulation
Rosy Chen, Mustafa Mukadam, Michael Kaess +4
Tactile dexterous manipulation is essential to automating complex household tasks, yet learning effective control policies remains a challenge. While recent work has relied on imit…
HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning
An Dang, Jayjun Lee, Mustafa Mukadam +4
In this paper, we address the problem of tactile sim-to-real policy transfer for contact-rich tasks. Existing methods primarily focus on vision-based sensors and emphasize image re…
TactAlign: Human-to-Robot Policy Transfer via Tactile Alignment
Youngsun Wi, Jessica Yin, Elvis Xiang +5
Human demonstrations collected by wearable devices (e.g., tactile gloves) provide fast and dexterous supervision for policy learning, and are guided by rich, natural tactile feedba…
Visuo-Tactile World Models
Carolina Higuera, Sergio Arnaud, Byron Boots +3
We introduce multi-task Visuo-Tactile World Models (VT-WM), which capture the physics of contact through touch reasoning. By complementing vision with tactile sensing, VT-WM better…
Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation
Carolina Higuera, Akash Sharma, Taosha Fan +8
We present Sparsh-X, the first multisensory touch representations across four tactile modalities: image, audio, motion, and pressure. Trained on ~1M contact-rich interactions colle…
Self-supervised perception for tactile skin covered dexterous hands
Akash Sharma, Carolina Higuera, Chaithanya Krishna Bodduluri +9
We present Sparsh-skin, a pre-trained encoder for magnetic skin sensors distributed across the fingertips, phalanges, and palm of a dexterous robot hand. Magnetic tactile skins off…
Geometric Retargeting: A Principled, Ultrafast Neural Hand Retargeting Algorithm
Zhao-Heng Yin, Changhao Wang, Luis Pineda +4
We introduce Geometric Retargeting (GeoRT), an ultrafast, and principled neural hand retargeting algorithm for teleoperation, developed as part of our recent Dexterity Gen (DexGen)…
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction
Huang Huang, Fangchen Liu, Letian Fu +5
Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions. Existing approaches require fine-tuning pre-trained visio…
DexterityGen: Foundation Controller for Unprecedented Dexterity
Zhao-Heng Yin, Changhao Wang, Luis Pineda +11
Teaching robots dexterous manipulation skills, such as tool use, presents a significant challenge. Current approaches can be broadly categorized into two strategies: human teleoper…
Sparsh: Self-supervised touch representations for vision-based tactile sensing
Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri +8
In this work, we introduce general purpose touch representations for the increasingly accessible class of vision-based tactile sensors. Such sensors have led to many recent advance…
A Touch, Vision, and Language Dataset for Multimodal Alignment
Letian Fu, Gaurav Datta, Huang Huang +7
Touch is an important sensing modality for humans, but it has not yet been incorporated into a multimodal generative language model. This is partially due to the difficulty of obta…
Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation
Sudharshan Suresh, Haozhi Qi, Tingfan Wu +9
To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, suc…
TaskMet: Task-Driven Metric Learning for Model Learning
Dishank Bansal, Ricky T. Q. Chen, Mustafa Mukadam +1
Deep learning models are often deployed in downstream tasks that the training procedure may not be aware of. For example, models solely trained to achieve accurate predictions may…
Perceiving Extrinsic Contacts from Touch Improves Learning Insertion Policies
Carolina Higuera, Joseph Ortiz, Haozhi Qi +3
Robotic manipulation tasks such as object insertion typically involve interactions between object and environment, namely extrinsic contacts. Prior work on Neural Contact Fields (N…
Decentralization and Acceleration Enables Large-Scale Bundle Adjustment
Taosha Fan, Joseph Ortiz, Ming Hsiao +4
Scaling to arbitrarily large bundle adjustment problems requires data and compute to be distributed across multiple devices. Centralized methods in prior works are only able to sol…
USA-Net: Unified Semantic and Affordance Representations for Robot Memory
Benjamin Bolte, Austin Wang, Jimmy Yang +3
In order for robots to follow open-ended instructions like "go open the brown cabinet over the sink", they require an understanding of both the scene geometry and the semantics of…
Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Models
Carolina Higuera, Byron Boots, Mustafa Mukadam
Simulating vision-based tactile sensors enables learning models for contact-rich tasks when collecting real world data at scale can be prohibitive. However, modeling the optical re…
Neural Grasp Distance Fields for Robot Manipulation
Thomas Weng, David Held, Franziska Meier +1
We formulate grasp learning as a neural field and present Neural Grasp Distance Fields (NGDF). Here, the input is a 6D pose of a robot end effector and output is a distance to a co…
MidasTouch: Monte-Carlo inference over distributions across sliding touch
Sudharshan Suresh, Zilin Si, Stuart Anderson +2
We present MidasTouch, a tactile perception system for online global localization of a vision-based touch sensor sliding on an object surface. This framework takes in posed tactile…
In-Hand Gravitational Pivoting Using Tactile Sensing
Jason Toskov, Rhys Newbury, Mustafa Mukadam +2
We study gravitational pivoting, a constrained version of in-hand manipulation, where we aim to control the rotation of an object around the grip point of a parallel gripper. To ac…
Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing
Carolina Higuera, Siyuan Dong, Byron Boots +1
We present Neural Contact Fields, a method that brings together neural fields and tactile sensing to address the problem of tracking extrinsic contact between object and environmen…