10 papers
ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery
Yufei Liu, Xixi Wang, Hao Li +8
Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-o…
Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills
Siyuan Huang, Pengyu Cheng, Haotian Liu +10
LLM training is shifting from manual design and annotation to interaction-driven self-evolution. However, existing self-evolutionary methods face a fundamental dilemma between task…
ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
Hao Li, Ganlong Zhao, Yufei Liu +8
Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show…
Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes
Tongyan Fang, Siyuan Huang, Naiyu Fang +6
When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome (success or failure), yet the actor update requires per-tr…
GDPO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization
Haotian Liu, Yihao Liu, Jingwei Ni +11
As LLMs advance, post-training reinforcement learning (RL) increasingly relies on multi-dimensional rewards to cultivate comprehensive capabilities. This shift demands new algorith…
Test-time Adversarial Takeover: A Real-time Hijacking Interface against Robotic Diffusion Policies
Zi Yin, Peilin Chai, Siyuan Huang +1
Diffusion-based action generation has become a foundational component of embodied AI, but its reliance on visual conditioning leaves deployed visuomotor policies vulnerable to adve…
Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
Tao Chen, Gangwei Jiang, Pengyu Cheng +10
Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines. However, current rew…
EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy
Jinzhao Li, Yinuo Chen, Dongxu Piao +9
Humans constantly reason about 3D proximity, the relations between their body and surrounding objects, to guide perception and action in daily life. Whether multimodal large langua…
Rethinking VLM Representation for VLA Initialization
Weifeng Lin, Siyuan Huang, Hao Li +5
Vision-Language-Action (VLA) models widely adopt pretrained Vision-Language Models (VLMs) as policy backbones, yet it remains unclear what kind of pretrained VLM representation is…
CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation
Xiaoqi Li, Lingyun Xu, Mingxu Zhang +8
In robotic, task goals can be conveyed through various modalities, such as language, goal images, and goal videos. However, natural language can be ambiguous, while images or video…