11 papers
Visual-Wi-Fi Navigation to Signal-Emitting Devices
Lihao Wang, Tianjian Xu, Tianmin Shu +1
Indoor navigation to task-relevant targets remains difficult when the target is small, occluded, or visually indistinct. Meanwhile, smart appliances, mobile devices, and IoT tags e…
Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination
Suyu Ye, Zheyuan Zhang, Vaishnav Tadiparthi +5
Robots operating in the physical world will increasingly need to coordinate with other robots, particularly in manipulation tasks where an object may be too large or heavy for a si…
JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shifts
Zheyuan Zhang, Suyu Ye, Nakul Agarwal +5
World models enable agents to plan by predicting future states of the environment, but their predictions can become unreliable when test-time dynamics differ from those seen during…
Continual Learning Mechanisms Compose for Long-Horizon Memorization
Zheyuan Zhang, Alvin Zhang, Daniel Khashabi +1
Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memoriz…
RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signals
Lihao Wang, Linlu Gao, Jiacan Yu +5
Recognizing Human-Object Interactions (HOI) is essential for intelligent systems, underpinning applications in virtual and augmented reality, embodied AI, and assistive robotics. H…
AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents
Zheyuan Zhang, Zehao Wen, Alvin Zhang +4
For agents to learn continuously from interaction with the world at test time, they must be able to explore effectively, acquire new world knowledge and skills, retain relevant epi…
3D-Belief: Embodied Belief Inference via Generative 3D World Modeling
Yifan Yin, Zehao Wen, Suyu Ye +10
Recent advances in visual generative models have highlighted the promise of learning generative world models. However, most existing approaches frame world modeling as novel-view s…
World-in-World: World Models in a Closed-Loop World
Jiahan Zhang, Muqing Jiang, Nanru Dai +14
Generative world models (WMs) can now simulate worlds with striking visual realism, which naturally raises the question of whether they can endow embodied agents with predictive pe…
PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation
Yifan Yin, Zhengtao Han, Shivam Aarya +6
Fine-grained robot manipulation, such as lifting and rotating a bottle to display the label on the cap, requires robust reasoning about object parts and their relationships with in…
GenEx: Generating an Explorable World
Taiming Lu, Tianmin Shu, Junfei Xiao +8
Understanding, navigating, and exploring the 3D physical real world has long been a central challenge in the development of artificial intelligence. In this work, we take a step to…
Generative World Explorer
Taiming Lu, Tianmin Shu, Alan Yuille +2
Planning with partial observation is a central challenge in embodied AI. A majority of prior works have tackled this challenge by developing agents that physically explore their en…