ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
arXiv:2304.03893 · doi:10.1109/ACCESS.2023.3310935
Abstract
This paper demonstrates how OpenAI's ChatGPT can be used in a few-shot setting to convert natural language instructions into a sequence of executable robot actions. The paper proposes easy-to-customize input prompts for ChatGPT that meet common requirements in practical applications, such as easy integration with robot execution systems and applicability to various environments while minimizing the impact of ChatGPT's token limit. The prompts encourage ChatGPT to output a sequence of predefined robot actions, represent the operating environment in a formalized style, and infer the updated state of the operating environment. Experiments confirmed that the proposed prompts enable ChatGPT to act according to requirements in various environments, and users can adjust ChatGPT's output with natural language feedback for safe and robust operation. The proposed prompts and source code are open-source and publicly available at https://github.com/microsoft/ChatGPT-Robot-Manipulation-Prompts
21 figures, 7 tables. Published in IEEE Access (in press). Last updated August 29th, 2023
References in corpus (13)
- Training language models to follow instructions with human feedback
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Evaluating Large Language Models Trained on Code
- Text2Motion: From Natural Language Instructions to Feasible Plans
- Inner Monologue: Embodied Reasoning through Planning with Language Models
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
- Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
- Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
- Translating Natural Language to Planning Goals with Large-Language Models
- Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting
- Grasp-type Recognition Leveraging Object Affordance
- Natural Language Robot Programming: NLP integrated with autonomous robotic grasping
- Remote Task-oriented Grasp Area Teaching By Non-Experts through Interactive Segmentation and Few-Shot Learning
Cited by in corpus (12)
- Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directions
- Real-World Robot Applications of Foundation Models: A Review
- GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration
- Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models
- CoPAL: Corrective Planning of Robot Actions with Large Language Models
- When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration
- A Framework for Neurosymbolic Robot Action Planning using Large Language Models
- Exploring Gen-AI applications in building research and industry: A review
- ChatSim: Underwater Simulation with Natural Language Prompting
- GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT
- When control meets large language models: From words to dynamics
- Open-Vocabulary Action Localization with Iterative Visual Prompting