8 papers
Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
Jiayi Fang
How should language interface with a world model's discrete symbol system? The dominant paradigm -- end-to-end injection of LLM/VLM features into robot world models (RT-2, Octo, Pa…
Emergent Semantic Representations in World Models through Physical Interaction without Linguistic Supervision
Jiayi Fang
What does a world model learn from physical exploration, without any linguistic supervision? We argue the answer is organized by a single principle: the geometric structure of the…
Embodiment Meets Environment: Toward Context-Aware, Safe Physical Caregiving Robots
Zhanxin Wu, Ruofei Tong, Jiaying Fang +1
Physical caregiving robots need to assist different users with different tasks in diverse environments, and they come in many embodiments. While substantial progress has been made…
Beyond Failure Recovery: An Engagement-Aware Human-in-the-loop Framework for Robotic Systems
Jiaying Fang, Joyce Yang, Zhanxin Wu +2
Conventional human-in-the-loop approaches typically involve users only when a robot encounters failure or uncertainty, treating humans primarily as tools for improving robot perfor…
Phantom: Training Robots Without Robots Using Only Human Videos
Marion Lepert, Jiaying Fang, Jeannette Bohg
Training general-purpose robots requires learning from large and diverse data sources. Current approaches rely heavily on teleoperated demonstrations which are difficult to scale.…
A Human-in-the-Loop Confidence-Aware Failure Recovery Framework for Modular Robot Policies
Rohan Banerjee, Krishna Palempalli, Bohan Yang +5
Robots operating in unstructured human environments inevitably encounter failures, especially in robot caregiving scenarios. While humans can often help robots recover, excessive o…