TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models

Abstract

TalkWithMachines aims to enhance human-robot interaction by contributing to interpretable industrial robotic systems, especially for safety-critical applications. The presented paper investigates recent advancements in Large Language Models (LLMs) and Vision Language Models (VLMs), in combination with robotic perception and control. This integration allows robots to understand and execute commands given in natural language and to perceive their environment through visual and/or descriptive inputs. Moreover, translating the LLM's internal states and reasoning into text that humans can easily understand ensures that operators gain a clearer insight into the robot's current state and intentions, which is essential for effective and safe operation. Our paper outlines four LLM-assisted simulated robotic control workflows, which explore (i) low-level control, (ii) the generation of language-based feedback that describes the robot's internal states, (iii) the use of visual information as additional input, and (iv) the use of robot structure information for generating task plans and feedback, taking the robot's physical capabilities and limitations into account. The proposed concepts are presented in a set of experiments, along with a brief discussion. Project description, videos, and supplementary materials will be available on the project website: https://talk-machines.github.io.

This paper has been accepted for publication in the proceedings of the 2024 Eighth IEEE International Conference on Robotic Computing (IRC)

References in corpus (15)

• ReAct: Synergizing Reasoning and Acting in Language Models
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
• Inner Monologue: Embodied Reasoning through Planning with Language Models
• Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic Review
• CLIPort: What and Where Pathways for Robotic Manipulation
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
• Language Models as Zero-Shot Trajectory Generators
• Large Language Models as General Pattern Machines
• Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis
• SayTap: Language to Quadrupedal Locomotion
• Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
• AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation
• RobotGPT: Robot Manipulation Learning from ChatGPT
• Human-Centric Autonomous Systems With LLMs for User Command Reasoning

Cited by in corpus (2)

• Keeping it Local, Tiny and Real: Automated Report Generation on Edge Computing Devices for Mechatronic-Based Cognitive Systems
• Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Models