LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models

Abstract

Task planning and motion planning are two of the most important problems in robotics, where task planning methods help robots achieve high-level goals and motion planning methods maintain low-level feasibility. Task and motion planning (TAMP) methods interleave the two processes of task planning and motion planning to ensure goal achievement and motion feasibility. Within the TAMP context, we are concerned with the mobile manipulation (MoMa) of multiple objects, where it is necessary to interleave actions for navigation and manipulation. In particular, we aim to compute where and how each object should be placed given underspecified goals, such as ``set up dinner table with a fork, knife and plate.'' We leverage the rich common sense knowledge from large language models (LLMs), e.g., about how tableware is organized, to facilitate both task-level and motion-level planning. In addition, we use computer vision methods to learn a strategy for selecting base positions to facilitate MoMa behaviors, where the base position corresponds to the robot's ``footprint'' and orientation in its operating space. Altogether, this article provides a principled TAMP framework for MoMa tasks that accounts for common sense about object rearrangement and is adaptive to novel situations that include many objects that need to be moved. We performed quantitative experiments in both real-world settings and simulated environments. We evaluated the success rate and efficiency in completing long-horizon object rearrangement tasks. While the robot completed 84.4\% real-world object rearrangement trials, subjective human evaluations indicated that the robot's performance is still lower than experienced human waiters.

References in corpus (21)

• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Scaling Laws for Neural Language Models
• Evaluating Large Language Models Trained on Code
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study
• Inner Monologue: Embodied Reasoning through Planning with Language Models
• TidyBot: Personalized Robot Assistance with Large Language Models
• FFRob: Leveraging Symbolic Planning for Efficient Task and Motion Planning
• DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
• LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
• Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worlds
• A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches
• Large Language Models as Commonsense Knowledge for Large-Scale Task Planning
• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
• MPTP: Motion-Planning-aware Task Planning for Navigation in Belief Space
• Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Following
• Multi-skill Mobile Manipulation for Object Rearrangement
• ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
• StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects
• DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planning