8 papers
Proxy Policy Steering
Chuanruo Ning, Tianrui Wang, Wei-Chiu Ma +1
Generalist robot policies carry broad manipulation priors from large-scale data, but specializing them to a new task remains the deployment bottleneck. This requires eliciting task…
Decoding Task Progress from VLA Representations
Atiksh Bhardwaj, Edward Weiyi Duan, Prithwish Dan +2
Vision-language-action models (VLAs) are moving rapidly towards deployment as general-purpose manipulation policies, but we currently lack basic tools for understanding what these…
X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations
Maximus A. Pace, Prithwish Dan, Chuanruo Ning +5
Human videos are a scalable source of training data for robot learning. However, humans and robots significantly differ in embodiment, making many human actions infeasible for dire…
Video Replanning via Latent Embedding Refinement and Rejection-Based Sampling
Po-Chen Ko, Yu Chan, Jiayuan Mao +7
Video planning has emerged as a flexible framework for robot manipulation, in which a generative model predicts a video of task completion, and a downstream module translates the p…
Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins
Chuanruo Ning, Kuan Fang, Wei-Chiu Ma
Recent advancements in open-world robot manipulation have been largely driven by vision-language models (VLMs). While these models exhibit strong generalization ability in high-lev…
X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
Prithwish Dan, Kushal Kedia, Angela Chao +4
Human videos offer a scalable way to train robot manipulation policies, but lack the action labels needed by standard imitation learning algorithms. Existing cross-embodiment appro…
Sampling-Based Grasp and Collision Prediction for Assisted Teleoperation
Simon Manschitz, Berk Gueler, Wei Ma +1
Shared autonomy allows for combining the global planning capabilities of a human operator with the strengths of a robot such as repeatability and accurate control. In a real-time t…
AVA: Attentive VLM Agent for Mastering StarCraft II
Weiyu Ma, Yuqian Fu, Zecheng Zhang +2
We introduce AVACraft, a multimodal StarCraft II benchmark supporting both Multi-Agent Reinforcement Learning (MARL) and Vision-Language Model (VLM) paradigms. Unlike SMAC-family e…