Publications (70)
Modality Forcing for Scalable Spatial Generation
Bardienus Pieter Duisterhof, Deva Ramanan, Jeffrey Ichnowski +2
Text-to-image (T2I) models contain rich spatial priors. Synthesizing photorealistic, cluttered scenes requires an understanding of geometry, including perspective and relative scal…
Intercepting the Future: Latent-Space Predictive World Model for Dynamic VLA Manipulation
Shahram Najam Syed, Arthur Jakobsson, Haoran Hao +1
Vision-Language-Action (VLA) models generalize across static manipulation but fail when objects move during task execution. They map the current observation to an action and assume…
Hearing the Slide: Acoustic-Guided Constraint Learning for Fast Non-Prehensile Transport
Yuemin Mao, Bardienus P. Duisterhof, Moonyoung Lee +1
Object transport tasks are fundamental in robotic automation, emphasizing the importance of efficient and secure methods for moving objects. Non-prehensile transport can significan…
Disentangling Dense Multi-Cable Knots
Vainavi Viswanath, Jennifer Grannen, Priya Sundaresan +7
Disentangling two or more cables requires many steps to remove crossings between and within cables. We formalize the problem of disentangling multiple cables and present an algorit…
Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision
Alberta Longhini, Marcel Büsching, Bardienus P. Duisterhof +4
We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dyna…
Functional Force-Aware Retargeting from Virtual Human Demos to Soft Robot Policies
Uksang Yoo, Mengjia Zhu, Evan Pezent +8
We introduce SoftAct, a framework for teaching soft robot hands to perform human-like manipulation skills by explicitly reasoning about contact forces. Leveraging immersive virtual…
Wiggle and Go! System Identification for Zero-Shot Dynamic Rope Manipulation
Arthur Jakobsson, Abhinav Mahajan, Karthik Pullalarevu +6
Many robotic tasks are unforgiving; a single mistake in a dynamic throw can lead to unacceptable delays or unrecoverable failure. To mitigate this, we present a novel approach that…
GOMP: Grasp-Optimized Motion Planning for Bin Picking
Jeffrey Ichnowski, Michael Danielczuk, Jingyi Xu +2
Rapid and reliable robot bin picking is a critical challenge in automating warehouses, often measured in picks-per-hour (PPH). We explore increasing PPH using faster motions based…
SonicBoom: Contact Localization Using Array of Microphones
Moonyoung Lee, Uksang Yoo, Jean Oh +3
In cluttered environments where visual sensors encounter heavy occlusion, such as in agricultural settings, tactile signals can provide crucial spatial information for the robot to…
Efficiently Calibrating Cable-Driven Surgical Robots with RGBD Fiducial Sensing and Recurrent Neural Networks
Minho Hwang, Brijen Thananjeyan, Samuel Paradis +5
Automation of surgical subtasks using cable-driven robotic surgical assistants (RSAs) such as Intuitive Surgical's da Vinci Research Kit (dVRK) is challenging due to imprecision in…
Automating Surgical Peg Transfer: Calibration with Deep Learning Can Exceed Speed, Accuracy, and Consistency of Humans
Minho Hwang, Jeffrey Ichnowski, Brijen Thananjeyan +5
Peg transfer is a well-known surgical training task in the Fundamentals of Laparoscopic Surgery (FLS). While human sur-geons teleoperate robots such as the da Vinci to perform this…
FogROS2-SGC: A ROS2 Cloud Robotics Platform for Secure Global Connectivity
Kaiyuan Chen, Ryan Hoque, Karthik Dharmarajan +5
The Robot Operating System (ROS2) is the most widely used software platform for building robotics applications. FogROS2 extends ROS2 to allow robots to access cloud computing on de…
Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction
Hongyi Chen, Tony Dong, Tiancheng Wu +7
Multi-finger robotic hand manipulation and grasping are challenging due to the high-dimensional action space and the difficulty of acquiring large-scale training data. Existing app…
Mechanical Search on Shelves using Lateral Access X-RAY
Huang Huang, Marcus Dominguez-Kuhne, Jeffrey Ichnowski +7
Efficiently finding an occluded object with lateral access arises in many contexts such as warehouses, retail, healthcare, shipping, and homes. We introduce LAX-RAY (Lateral Access…
Untangling Dense Knots by Learning Task-Relevant Keypoints
Jennifer Grannen, Priya Sundaresan, Brijen Thananjeyan +7
Untangling ropes, wires, and cables is a challenging task for robots due to the high-dimensional configuration space, visual homogeneity, self-occlusions, and complex dynamics. We…
Accelerating Quadratic Optimization with Reinforcement Learning
Jeffrey Ichnowski, Paras Jain, Bartolomeo Stellato +6
First-order methods for quadratic optimization such as OSQP are widely used for large-scale machine learning and embedded optimal control, where many related problems must be rapid…
Learning to Efficiently Plan Robust Frictional Multi-Object Grasps
Wisdom C. Agboh, Satvik Sharma, Kishore Srinivas +7
We consider a decluttering problem where multiple rigid convex polygonal objects rest in randomly placed positions and orientations on a planar surface and must be efficiently tran…
Soft Robotic Dynamic In-Hand Pen Spinning
Yunchao Yao, Uksang Yoo, Jean Oh +2
Dynamic in-hand manipulation remains a challenging task for soft robotic systems that have demonstrated advantages in safe compliant interactions but struggle with high-speed dynam…
RoPotter: Toward Robotic Pottery and Deformable Object Manipulation with Structural Priors
Uksang Yoo, Adam Hung, Jonathan Francis +2
Humans are capable of continuously manipulating a wide variety of deformable objects into complex shapes. This is made possible by our intuitive understanding of material propertie…
KineSoft: Learning Proprioceptive Manipulation Policies with Soft Robot Hands
Uksang Yoo, Jonathan Francis, Jean Oh +1
Underactuated soft robot hands offer inherent safety and adaptability advantages over rigid systems, but developing dexterous manipulation skills remains challenging. While imitati…
Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts
Hongyi Chen, Yunchao Yao, Ruixuan Liu +2
Current robot autonomy struggles to operate beyond the assumed Operational Design Domain (ODD), the specific set of conditions and environments in which the system is designed to f…
Residual-NeRF: Learning Residual NeRFs for Transparent Object Manipulation
Bardienus P. Duisterhof, Yuemin Mao, Si Heng Teng +1
Transparent objects are ubiquitous in industry, pharmaceuticals, and households. Grasping and manipulating these objects is a significant challenge for robots. Existing methods hav…
A-SLIP: Acoustic Sensing for Continuous In-hand Slip Estimation
Uksang Yoo, Yuemin Mao, Jean Oh +1
Reliable in-hand manipulation requires accurate real-time estimation of slip between a gripper and a grasped object. Existing tactile sensing approaches based on vision, capacitanc…
VibeAct: Vibration to Actions for Contact-Rich Reactive Robot Dexterity
Yuemin Mao, Uksang Yoo, Jean Oh +2
Dexterous manipulation depends on contact events that are fast, local, and often visually occluded. Piezoelectric microphones offer a compact and high-bandwidth way to sense these…
SOFTMAP: Sim2Real Soft Robot Forward Modeling via Topological Mesh Alignment and Physics Prior
Ziyong Ma, Uksang Yoo, Jonathan Francis +3
While soft robot manipulators offer compelling advantages over rigid counterparts, including inherent compliance, safe human-robot interaction, and the ability to conform to comple…
Visuo-Acoustic Hand Pose and Contact Estimation
Yuemin Mao, Uksang Yoo, Yunchao Yao +5
Accurately estimating hand pose and hand-object contact events is essential for robot data-collection, immersive virtual environments, and biomechanical analysis, yet remains chall…
Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor
Daniel Seita, Aditya Ganapathi, Ryan Hoque +11
Sequential pulling policies to flatten and smooth fabrics have applications from surgery to manufacturing to home tasks such as bed making and folding clothes. Due to the complexit…
Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions
Hongyi Chen, Yunchao Yao, Yufei Ye +8
Functional grasping is essential for enabling dexterous multi-finger robot hands to manipulate objects effectively. Prior work largely focuses on power grasps, which only involve h…
Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
Jeffrey Ichnowski, Yahav Avigal, Justin Kerr +1
The ability to grasp and manipulate transparent objects is a major challenge for robots. Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of…
BOMP: Bin-Optimized Motion Planning
Zachary Tam, Karthik Dharmarajan, Tianshuang Qiu +3
In logistics, the ability to quickly compute and execute pick-and-place motions from bins is critical to increasing productivity. We present Bin-Optimized Motion Planning (BOMP), a…
ORB: Operating Room Bot, Automating Operating Room Logistics through Mobile Manipulation
Jinkai Qiu, Yungjun Kim, Gaurav Sethia +4
Efficiently delivering items to an ongoing surgery in a hospital operating room can be a matter of life or death. In modern hospital settings, delivery robots have successfully tra…
Adversarial Game-Theoretic Algorithm for Dexterous Grasp Synthesis
Yu Chen, Botao He, Yuemin Mao +7
For many complex tasks, multi-finger robot hands are poised to revolutionize how we interact with the world, but reliably grasping objects remains a significant challenge. We focus…
Inclusion in Assistive Haircare Robotics: Practical and Ethical Considerations in Hair Manipulation
Uksang Yoo, Nathaniel Dennler, Sarvesh Patil +2
Robot haircare systems could provide a controlled and personalized environment that is respectful of an individual's sensitivities and may offer a comfortable experience. We argue…
Mechanical Search on Shelves with Efficient Stacking and Destacking of Objects
Huang Huang, Letian Fu, Michael Danielczuk +6
Stacking increases storage efficiency in shelves, but the lack of visibility and accessibility makes the mechanical search problem of revealing and extracting target objects diffic…
RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion
Bardienus P. Duisterhof, Jan Oberst, Bowen Wen +3
3D shape completion has broad applications in robotics, digital twin reconstruction, and extended reality (XR). Although recent advances in 3D object and scene completion have achi…
Planar Robot Casting with Real2Sim2Real Self-Supervised Learning
Vincent Lim, Huang Huang, Lawrence Yunliang Chen +5
This paper introduces the task of {\em Planar Robot Casting (PRC)}: where one planar motion of a robot arm holding one end of a cable causes the other end to slide across the plane…
FogROS2-FT: Fault Tolerant Cloud Robotics
Kaiyuan Chen, Kush Hari, Trinity Chung +8
Cloud robotics enables robots to offload complex computational tasks to cloud servers for performance and ease of management. However, cloud compute can be costly, cloud services c…
FogROS2: An Adaptive Platform for Cloud and Fog Robotics Using ROS 2
Jeffrey Ichnowski, Kaiyuan Chen, Karthik Dharmarajan +12
Mobility, power, and price points often dictate that robots do not have sufficient computing power on board to run contemporary robot algorithms at desired rates. Cloud computing p…
SemAnCorr: Semantic Anchored Correspondence for Zero-Shot Manipulation Skill Transfer
Xiaoxiang Dong, William Baron, Hongyi Chen +3
The paper presents SemAnCorr, a training‑free method that uses semantically consistent anchor regions and functional maps to create dense, geometrically coherent correspondences be…
Self-Supervised Learning of Dynamic Planar Manipulation of Free-End Cables
Jonathan Wang, Huang Huang, Vincent Lim +5
Dynamic manipulation of free-end cables has applications for cable management in homes, warehouses and manufacturing plants. We present a supervised learning approach for dynamic m…
Kit-Net: Self-Supervised Learning to Kit Novel 3D Objects into Novel 3D Cavities
Shivin Devgon, Jeffrey Ichnowski, Michael Danielczuk +6
In industrial part kitting, 3D objects are inserted into cavities for transportation or subsequent assembly. Kitting is a critical step as it can decrease downstream processing and…
Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation
Samuel Paradis, Minho Hwang, Brijen Thananjeyan +6
Automation of surgical tasks using cable-driven robots is challenging due to backlash, hysteresis, and cable tension, and these issues are exacerbated as surgical instruments must…
Mechanical Search on Shelves using a Novel "Bluction" Tool
Huang Huang, Michael Danielczuk, Chung Min Kim +6
Shelves are common in homes, warehouses, and commercial settings due to their storage efficiency. However, this efficiency comes at the cost of reduced visibility and accessibility…
Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies
Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan +7
Robot manipulation for untangling 1D deformable structures such as ropes, cables, and wires is challenging due to their infinite dimensional configuration space, complex dynamics,…
FogROS2-Config: Optimizing Latency and Cost for Multi-Cloud Robot Applications
Kaiyuan Chen, Kush Hari, Rohil Khare +6
Cloud service providers provide over 50,000 distinct and dynamically changing set of cloud server options. To help roboticists make cost-effective decisions, we present FogROS2-Con…
POE: Acoustic Soft Robotic Proprioception for Omnidirectional End-effectors
Uksang Yoo, Ziven Lopez, Jeffrey Ichnowski +1
Soft robotic shape estimation and proprioception are challenging because of soft robot's complex deformation behaviors and infinite degrees of freedom. A soft robot's continuously…
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
Justin Kerr, Huang Huang, Albert Wilcox +4
Humans make extensive use of vision and touch as complementary senses, with vision providing global information about the scene and touch measuring local information during manipul…
Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms
Shivin Devgon, Jeffrey Ichnowski, Ashwin Balakrishna +2
Orienting objects is a critical component in the automation of many packing and assembly tasks. We present an algorithm to orient novel objects given a depth image of the object in…
SGTM 2.0: Autonomously Untangling Long Cables using Interactive Perception
Kaushik Shivakumar, Vainavi Viswanath, Anrui Gu +6
Cables are commonplace in homes, hospitals, and industrial warehouses and are prone to tangling. This paper extends prior work on autonomously untangling long cables by introducing…
LEGS: Learning Efficient Grasp Sets for Exploratory Grasping
Letian Fu, Michael Danielczuk, Ashwin Balakrishna +4
While deep learning has enabled significant progress in designing general purpose robot grasping systems, there remain objects which still pose challenges for these systems. Recent…
Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles
Albert Wilcox, Justin Kerr, Brijen Thananjeyan +5
Robotic Surgical Assistants (RSAs) are commonly used to perform minimally invasive surgeries by expert surgeons. However, long procedures filled with tedious and repetitive tasks s…
FogROS G: Enabling Secure, Connected and Mobile Fog Robotics with Global Addressability
Kaiyuan Chen, Jiachen Yuan, Nikhil Jha +3
Fog Robotics renders networked robots with greater mobility, on-demand compute capabilities and better energy efficiency by offloading heavy robotics workloads to nearby Edge and d…
DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation
Bardienus P. Duisterhof, Zhao Mandi, Yunchao Yao +8
Teaching robots to fold, drape, or reposition deformable objects such as cloth will unlock a variety of automation applications. While remarkable progress has been made for rigid o…
KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation
Hongyi Chen, Abulikemu Abuduweili, Aviral Agrawal +4
Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between objects and multi-fingered hands. Ko…
Policy-Based Bayesian Experimental Design for Non-Differentiable Implicit Models
Vincent Lim, Ellen Novoseller, Jeffrey Ichnowski +2
For applications in healthcare, physics, energy, robotics, and many other fields, designing maximally informative experiments is valuable, particularly when experiments are expensi…
Efficiently Learning Single-Arm Fling Motions to Smooth Garments
Lawrence Yunliang Chen, Huang Huang, Ellen Novoseller +6
Recent work has shown that 2-arm "fling" motions can be effective for garment smoothing. We consider single-arm fling motions. Unlike 2-arm fling motions, which require little robo…
Soft and Compliant Contact-Rich Hair Manipulation and Care
Uksang Yoo, Nathaniel Dennler, Eliot Xing +4
Hair care robots can help address labor shortages in elderly care while enabling those with limited mobility to maintain their hair-related identity. We present MOE-Hair, a soft ro…
DYMO-Hair: Generalizable Volumetric Dynamics Modeling for Robot Hair Manipulation
Chengyang Zhao, Uksang Yoo, Arkadeep Narayan Chaudhury +4
Hair care is an essential daily activity, yet it remains inaccessible to individuals with limited mobility and challenging for autonomous robot systems due to the fine-grained phys…
GOMP-FIT: Grasp-Optimized Motion Planning for Fast Inertial Transport
Jeffrey Ichnowski, Yahav Avigal, Yi Liu +1
High-speed motions in pick-and-place operations are critical to making robots cost-effective in many automation scenarios, from warehouses and manufacturing to hospitals and homes.…
FogROS: An Adaptive Framework for Automating Fog Robotics Deployment
Kaiyuan, Chen, Yafei Liang +6
As many robot automation applications increasingly rely on multi-core processing or deep-learning models, cloud computing is becoming an attractive and economically viable resource…
Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot
Minho Hwang, Daniel Seita, Brijen Thananjeyan +5
Recent advances in depth-sensing have significantly increased accuracy, resolution, and frame rate, as shown in the 1920x1200 resolution and 13 frames per second Zivid RGBD camera.…
Multi-Object Grasping in the Plane
Wisdom C. Agboh, Jeffrey Ichnowski, Ken Goldberg +1
We consider a novel problem where multiple rigid convex polygonal objects rest in randomly placed positions and orientations on a planar surface visible from an overhead camera. Th…
Robots of the Lost Arc: Self-Supervised Learning to Dynamically Manipulate Fixed-Endpoint Cables
Harry Zhang, Jeffrey Ichnowski, Daniel Seita +3
We explore how high-speed robot arm motions can dynamically manipulate cables to vault over obstacles, knock objects from pedestals, and weave between obstacles. In this paper, we…
3PoinTr: 3D Point Tracks for Learning Manipulation from Unconstrained Human Videos
Adam Hung, Bardienus Pieter Duisterhof, Jeffrey Ichnowski
Learning manipulation policies from human videos could greatly reduce the need for expensive robot demonstrations, but existing approaches typically require restrictive assumptions…
The Teenager's Problem: Efficient Garment Decluttering as Probabilistic Set Cover
Aviv Adler, Ayah Ahmad, Yulei Qiu +9
This paper addresses the "Teenager's Problem": efficiently removing scattered garments from a planar surface into a basket. As grasping and transporting individual garments is high…
GOMP-ST: Grasp Optimized Motion Planning for Suction Transport
Yahav Avigal, Jeffrey Ichnowski, Max Yiye Cao +1
Suction cup grasping is very common in industry, but moving too quickly can cause suction cups to detach, causing drops or damage. Maintaining a suction grasp throughout a high-spe…
Optimal Shelf Arrangement to Minimize Robot Retrieval Time
Lawrence Yunliang Chen, Huang Huang, Michael Danielczuk +2
Shelves are commonly used to store objects in homes, stores, and warehouses. We formulate the problem of Optimal Shelf Arrangement (OSA), where the goal is to optimize the arrangem…
FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement
Haoran Hao, Shahram Najam Syed, Jeffrey Ichnowski +1
Robot policies inevitably encounter failures when deployed in real environments. Naive retries often repeat the same mistakes, while many existing recovery methods rely on human in…
HANDLOOM: Learned Tracing of One-Dimensional Objects for Inspection and Manipulation
Vainavi Viswanath, Kaushik Shivakumar, Jainil Ajmera +6
Tracing - estimating the spatial state of - long deformable linear objects such as cables, threads, hoses, or ropes, is useful for a broad range of tasks in homes, retail, factorie…
Autonomously Untangling Long Cables
Vainavi Viswanath, Kaushik Shivakumar, Justin Kerr +7
Cables are ubiquitous in many settings and it is often useful to untangle them. However, cables are prone to self-occlusions and knots, making them difficult to perceive and manipu…