Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

Abstract

Large-language models (LLMs) hold significant promise in improving human-robot interaction, offering advanced conversational skills and versatility in managing diverse, open-ended user requests in various tasks and domains. Despite the potential to transform human-robot interaction, very little is known about the distinctive design requirements for utilizing LLMs in robots, which may differ from text and voice interaction and vary by task and context. To better understand these requirements, we conducted a user study (n = 32) comparing an LLM-powered social robot against text- and voice-based agents, analyzing task-based requirements in conversational tasks, including choose, generate, execute, and negotiate. Our findings show that LLM-powered robots elevate expectations for sophisticated non-verbal cues and excel in connection-building and deliberation, but fall short in logical communication and may induce anxiety. We provide design implications both for robots integrating LLMs and for fine-tuning LLMs for use with robots.

arXiv comment10 pages, 4 figures. Callie Y. Kim and Christine P. Lee contributed equally to the work. To be published in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI '24), March 11--14, 2024, Boulder, CO, USA

References in corpus (12)

• Survey of Hallucination in Natural Language Generation
• Training language models to follow instructions with human feedback
• LoRA: Low-Rank Adaptation of Large Language Models
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• PaLM-E: An Embodied Multimodal Language Model
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Embodiment in Socially Interactive Robots
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback
• Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models
• ChatGPT for Robotics: Design Principles and Model Abilities
• OpenAGI: When LLM Meets Domain Experts
• Developing Social Robots with Empathetic Non-Verbal Cues Using Large Language Models

Cited by in corpus (10)

• The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare
• SET-PAiREd: Designing for Parental Involvement in Learning with an AI-Assisted Educational Robot
• To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions
• Bridging Generations using AI-Supported Co-Creative Activities
• The AI-DEC: A Card-based Design Method for User-centered AI Explanations
• "This really lets us see the entire world:" Designing a conversational telepresence robot for homebound older adults
• GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication
• Are Large Language Models Aligned with People's Social Intuitions for Human-Robot Interactions?
• Word Synchronization Challenge: A Benchmark for Word Association Responses for Large Language Models
• RoboCritics: Enabling Reliable End-to-End LLM Robot Programming through Expert-Informed Critics