ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning

Abstract

We present a framework for intuitive robot programming by non-experts, leveraging natural language prompts and contextual information from the Robot Operating System (ROS). Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface. Key features of the framework include: integration of ROS with an AI agent connected to a plethora of open-source and commercial LLMs, automatic extraction of a behavior from the LLM output and execution of ROS actions/services, support for three behavior modes (sequence, behavior tree, state machine), imitation learning for adding new robot actions to the library of possible actions, and LLM reflection via human and environment feedback. Extensive experiments validate the framework, showcasing robustness, scalability, and versatility in diverse scenarios, including long-horizon tasks, tabletop rearrangements, and remote supervisory control. To facilitate the adoption of our framework and support the reproduction of our results, we have made our code open-source. You can access it at: https://github.com/huawei-noah/HEBO/tree/master/ROSLLM.

This document contains 26 pages and 13 figures

References in corpus (10)

• Learning agile and dynamic motor skills for legged robots
• Robot Operating System 2: Design, Architecture, and Uses In The Wild
• Behavior Trees in Robotics and AI: An Introduction
• Behavior Trees and State Machines in Robotics Applications
• LaMI: Large Language Models for Multi-Modal Human-Robot Interaction
• Language Models as Zero-Shot Trajectory Generators
• BTGenBot: Behavior Tree Generation for Robotic Tasks with Lightweight LLMs
• To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions
• The Robot Economy: Here It Comes
• Multimodal Coherent Explanation Generation of Robot Failures