8 papers
ReflectWorld-MM: An Entity-Oriented Multimodal Memory System for Open-Ended Video Streams
Xiaokang Ma, Yifan Sun, Zhihong Jin +6
Building assistants that can continually watch the world, remember what they see, and reason over their accumulated experience is a long-standing goal, and recently multimodal agen…
DeformMaster: An Interactive Physics-Neural World Model for Deformable Objects from Videos
Can Li, Zhoujian Li, Ren Li +4
World models for deformable objects should recover not only geometry and appearance, but also underlying physical dynamics, interaction grounding, and material behavior. Learning s…
Move-Then-Operate: Behavioral Phasing for Human-Like Robotic Manipulation
Haoming Xu, Lei Lei, Jie Gu +3
We present Move-Then-Operate, a Vision language action framework that explicitly decouples robotic manipulation into two distinct behavioral phases: coarse relocation (move) and co…
Towards Dexterous Embodied Manipulation via Deep Multi-Sensory Fusion and Sparse Expert Scaling
Yirui Sun, Guangyu Zhuge, Keliang Liu +5
Realizing dexterous embodied manipulation necessitates the deep integration of heterogeneous multimodal sensory inputs. However, current vision-centric paradigms often overlook the…
Gaussian Sequences with Multi-Scale Dynamics for 4D Reconstruction from Monocular Casual Videos
Can Li, Jie Gu, Jingmin Chen +2
Understanding dynamic scenes from casual videos is critical for scalable robot learning, yet four-dimensional (4D) reconstruction under strictly monocular settings remains highly i…
Egocentric Instruction-oriented Affordance Prediction via Large Multimodal Model
Bokai Ji, Jie Gu, Xiaokang Ma +3
Affordance is crucial for intelligent robots in the context of object manipulation. In this paper, we argue that affordance should be task-/instruction-dependent, which is overlook…
Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective
Lei Lei, Jie Gu, Xiaokang Ma +3
Existing Multimodal Large Language Models (MLLMs) process a large number of visual tokens, leading to significant computational costs and inefficiency. Instruction-related visual t…
Stimulating Imagination: Towards General-purpose "Something Something Placement"
Jianyang Wu, Jie Gu, Xiaokang Ma +3
General-purpose object placement is a fundamental capability of an intelligent generalist robot: being capable of rearranging objects following precise human instructions even in n…