When Robots Get Chatty: Grounding Multimodal Human-Robot Conversation and Collaboration

Abstract

We investigate the use of Large Language Models (LLMs) to equip neural robotic agents with human-like social and cognitive competencies, for the purpose of open-ended human-robot conversation and collaboration. We introduce a modular and extensible methodology for grounding an LLM with the sensory perceptions and capabilities of a physical robot, and integrate multiple deep learning models throughout the architecture in a form of system integration. The integrated models encompass various functions such as speech recognition, speech generation, open-vocabulary object detection, human pose estimation, and gesture detection, with the LLM serving as the central text-based coordinating unit. The qualitative and quantitative results demonstrate the huge potential of LLMs in providing emergent cognition and interactive language-oriented control of robots in a natural and social manner.

References in corpus (7)

• Learning Transferable Visual Models From Natural Language Supervision
• ReAct: Synergizing Reasoning and Acting in Language Models
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
• ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
• LaMI: Large Language Models for Multi-Modal Human-Robot Interaction
• The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs

Cited by in corpus (2)

• A Survey on State-of-the-art Deep Learning Applications and Challenges
• Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task