5 papers
TAP-VLA: Tactile Annotation Prompting for Vision Language Action Models
Mark Van der Merwe, Mohamad Louai Shehab, Jayjun Lee +4
Vision-Language-Action (VLA) models demonstrate impressive reasoning over visual, semantic, and spatial task variations by leveraging large-scale vision and language pre-training.…
Estimating Deformable-Rigid Contact Interactions for a Deformable Tool via Learning and Model-Based Optimization
Mark Van der Merwe, Miquel Oller, Dmitry Berenson +1
Dexterous manipulation requires careful reasoning over extrinsic contacts. The prevalence of deforming tools in human environments, the use of deformable sensors, and the increasin…
This&That: Language-Gesture Controlled Video Generation for Robot Planning
Boyang Wang, Nikhil Sridhar, Chao Feng +4
Clear, interpretable instructions are invaluable when attempting any complex task. Good instructions help to clarify the task and even anticipate the steps needed to solve it. In t…
Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback
Mark Van der Merwe, Youngsun Wi, Dmitry Berenson +1
Reasoning over the interplay between object deformation and force transmission through contact is central to the manipulation of compliant objects. In this paper, we propose Neural…
Machine Vision based Sample-Tube Localization for Mars Sample Return
Shreyansh Daftry, Barry Ridge, William Seto +8
A potential Mars Sample Return (MSR) architecture is being jointly studied by NASA and ESA. As currently envisioned, the MSR campaign consists of a series of 3 missions: sample cac…