18 papers
StateCraft: Augmenting Video World Models with Abstract-State Reasoning
Lorenzo Venturoli, Nedko Savov, Xi Wang +2
Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and un…
DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents
A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong +2
Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually similar controls compete for at…
ChronoGraph: Functional 4D Scene Graphs with Vision-Language Models for Interaction Understanding and Grounded Planning
Chenyangguang Zhang, Malgorzata Gwiazda, Guanlong Jiao +5
Embodied agents must determine where to act, anticipate the resulting scene changes, and interpret observed outcomes to guide subsequent actions. This requires connecting 4D intera…
Long Time No See: Benchmarking VLMs for Out-of-Sight Spatiotemporal Reasoning in Egocentric Videos
Fangzhou Ma, Ivo Alexander Ban, Eren Homburg +5
Real-world AI systems must reason about objects that are no longer visible: an AR assistant guiding a user back to an object used earlier, a household robot retrieving an item some…
EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal
Alex Brandes, Haig Conti Georges Sajelian, Manthan Patel +11
Deploying humanoid robots in unstructured terrain remains an open problem. While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more…
CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views
Alexey Gavryushin, Dingxi Zhang, Zhao Huang +8
Human-human collaboration is a fundamental aspect of everyday life, essential to success in a wide range of goal-directed activities from household tasks to professional teamwork.…
Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
Jiuming Liu, Chaojun Ni, Mengmeng Liu +7
With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream do…
From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data
Zhiyuan Feng, Qixiu Li, Huizhi Liang +12
Recent progress in generalizable embodied control has been driven by large-scale pretraining of Vision-Language-Action (VLA) models. However, most existing approaches rely on large…
World Model for Robot Learning: A Comprehensive Survey
Bohan Hou, Gen Li, Jindou Jia +15
World models, which are predictive representations of how environments evolve under actions, have become a central component of robot learning. They support policy learning, planni…
Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models
Marcel Gröpl, Jaewoo Jung, Seungryong Kim +2
Despite rapid progress, pretrained vision-language models still struggle when answers depend on tiny visual details or on combining clues spread across multiple regions, as in docu…
EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World
Ryan Punamiya, Simar Kareer, Zeyi Liu +37
Robot learning increasingly depends on large and diverse data, yet robot data collection remains expensive and difficult to scale. Egocentric human data offer a promising alternati…
AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding
Haozhe Qi, Kevin Qu, Mahdi Rad +3
Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by sco…
CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling
Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik +4
Video Language Models (VideoLMs) enable AI systems to understand temporal dynamics in videos. To fit within the maximum context window constraint, current methods use keyframe samp…
Training-free Detection and 6D Pose Estimation of Unseen Surgical Instruments
Jonas Hein, Lilian Calvet, Matthias Seibold +3
Purpose: Accurate detection and 6D pose estimation of surgical instruments are crucial for many computer-assisted interventions. However, supervised methods lack flexibility for ne…
Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints
Chenyangguang Zhang, Botao Ye, Boqi Chen +4
Controllable video generation for complex hand-object interactions is a critical step toward building visual world models. However, existing methods often struggle to achieve fine-…
Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents
Reuben Tan, Baolin Peng, Zhengyuan Yang +15
Agentic reasoning models trained with multimodal reinforcement learning (MMRL) have become increasingly capable, yet they are almost universally optimized using sparse, outcome-bas…
RocSync: Millisecond-Accurate Temporal Synchronization for Heterogeneous Camera Systems
Jaro Meyer, Frédéric Giraud, Joschua Wüthrich +3
Accurate spatiotemporal alignment of multi-view video streams is essential for a wide range of dynamic-scene applications such as multi-view 3D reconstruction, pose estimation, and…
Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model
Ruiping Liu, Junwei Zheng, Yufan Chen +6
Physical environments and circumstances are fundamentally dynamic, yet current 3D datasets and evaluation benchmarks tend to concentrate on either dynamic scenarios or dynamic situ…