8 papers
Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
Jiayi Fang
How should language interface with a world model's discrete symbol system? The dominant paradigm -- end-to-end injection of LLM/VLM features into robot world models (RT-2, Octo, Pa…
Embodiment Meets Environment: Toward Context-Aware, Safe Physical Caregiving Robots
Zhanxin Wu, Ruofei Tong, Jiaying Fang +1
Physical caregiving robots need to assist different users with different tasks in diverse environments, and they come in many embodiments. While substantial progress has been made…
Beyond Failure Recovery: An Engagement-Aware Human-in-the-loop Framework for Robotic Systems
Jiaying Fang, Joyce Yang, Zhanxin Wu +2
Conventional human-in-the-loop approaches typically involve users only when a robot encounters failure or uncertainty, treating humans primarily as tools for improving robot perfor…
Emergent Semantic Representations in World Models through Physical Interaction without Linguistic Supervision
Jiayi Fang
What does a world model learn from physical exploration, without any linguistic supervision? We argue the answer is organized by a single principle: the geometric structure of the…
A Human-in-the-Loop Confidence-Aware Failure Recovery Framework for Modular Robot Policies
Rohan Banerjee, Krishna Palempalli, Bohan Yang +5
Robots operating in unstructured human environments inevitably encounter failures, especially in robot caregiving scenarios. While humans can often help robots recover, excessive o…
Masquerade: Learning from In-the-wild Human Videos using Data-Editing
Marion Lepert, Jiaying Fang, Jeannette Bohg
Robot manipulation research still suffers from significant data scarcity: even the largest robot datasets are orders of magnitude smaller and less diverse than those that fueled re…
Phantom: Training Robots Without Robots Using Only Human Videos
Marion Lepert, Jiaying Fang, Jeannette Bohg
Training general-purpose robots requires learning from large and diverse data sources. Current approaches rely heavily on teleoperated demonstrations which are difficult to scale.…
Force-Aware Autonomous Robotic Surgery
Alaa Eldin Abdelaal, Jiaying Fang, Tim N. Reinhart +4
This work demonstrates the benefits of using tool-tissue interaction forces in the design of autonomous systems in robot-assisted surgery (RAS). Autonomous systems in surgery must…