A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot

Abstract

Despite recent advances in integrating Large Language Models (LLMs) into social robotics, two weaknesses persist. First, existing implementations on platforms like Pepper often rely on cascaded Speech-to-Text (STT)->LLM->Text-to-Speech (TTS) pipelines, resulting in high latency and the loss of paralinguistic information. Second, most implementations fail to fully leverage the LLM's capabilities for multimodal perception and agentic control. We present an open-source Android framework for the Pepper robot that addresses these limitations through two key innovations. First, we integrate end-to-end Speech-to-Speech (S2S) models to achieve low-latency interaction while preserving paralinguistic cues and enabling adaptive intonation. Second, we implement extensive Function Calling capabilities that elevate the LLM to an agentic planner, orchestrating robot actions (navigation, gaze control, tablet interaction) and integrating diverse multimodal feedback (vision, touch, system state). The framework runs on the robot's tablet but can also be built to run on regular Android smartphones or tablets, decoupling development from robot hardware. This work provides the HRI community with a practical, extensible platform for exploring advanced LLM-driven embodied interaction.

arXiv comment4 pages, 2 figures. To appear in Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction (HRI '26), Edinburgh, Scotland, March 2026

References in corpus (11)

• A Survey on Integration of Large Language Models with Intelligent Robots
• I Was Blind but Now I See: Implementing Vision-Enabled Dialogue in Social Robots
• Touch in Human Social Robot Interaction: Systematic Literature Review with PRISMA Method
• A Sign Language Recognition System with Pepper, Lightweight-Transformer, and LLM
• Does ChatGPT and Whisper Make Humanoid Robots More Relatable?
• Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction
• HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interaction
• MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols
• A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment
• Towards Multimodal Social Conversations with Robots: Using Vision-Language Models
• Direct Speech to Speech Translation: A Review