Text2Motion: From Natural Language Instructions to Feasible Plans

Abstract

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals. Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of partial affordance perception. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 82%, while prior state-of-the-art language-based planning methods only achieve 13%. Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between skills.

arXiv commentPublished in Autonomous Robots, Special Issue: Large Language Models in Robotics 2023. Project page: https://sites.google.com/stanford.edu/text2motion. First two authors contributed equally

References in corpus (14)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• The Fast Downward Planning System
• TidyBot: Personalized Robot Assistance with Large Language Models
• CLIPort: What and Where Pathways for Robotic Manipulation
• ChatGPT for Robotics: Design Principles and Model Abilities
• BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
• Representation, learning, and planning algorithms for geometric task and motion planning
• Language-Conditioned Imitation Learning for Robot Manipulation Tasks
• Learning Multi-Object Dynamics with Compositional Neural Radiance Fields
• STAP: Sequencing Task-Agnostic Policies
• Learning Neuro-Symbolic Skills for Bilevel Planning
• CAMPs: Learning Context-Specific Abstractions for Efficient Planning in Factored MDPs

Cited by in corpus (15)

• TidyBot: Personalized Robot Assistance with Large Language Models
• A Survey on Integration of Large Language Models with Intelligent Robots
• ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
• Real-World Robot Applications of Foundation Models: A Review
• GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration
• Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation
• Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worlds
• A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches
• On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)
• CoPAL: Corrective Planning of Robot Actions with Large Language Models
• DISCO: Language-Guided Manipulation with Diffusion Policies and Constrained Inpainting
• Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback
• When control meets large language models: From words to dynamics
• Prime the search: Using large language models for guiding geometric task and motion planning by warm-starting tree search
• Incremental Language Understanding for Online Motion Planning of Robot Manipulators