World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models
arXiv:2607.27599
The paper introduces World Action Planner, a robot planning system that combines vision‑language models with a multi‑task, pose‑image conditioned world model to generate and iteratively refine action plans, achieving strong zero‑shot and compositional generalization.
Abstract
Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io
Project page at worldactionplanner.github.io