Incremental Language Understanding for Online Motion Planning of Robot Manipulators

Abstract

Human-robot interaction requires robots to process language incrementally, adapting their actions in real-time based on evolving speech input. Existing approaches to language-guided robot motion planning typically assume fully specified instructions, resulting in inefficient stop-and-replan behavior when corrections or clarifications occur. In this paper, we introduce a novel reasoning-based incremental parser which integrates an online motion planning algorithm within the cognitive architecture. Our approach enables continuous adaptation to dynamic linguistic input, allowing robots to update motion plans without restarting execution. The incremental parser maintains multiple candidate parses, leveraging reasoning mechanisms to resolve ambiguities and revise interpretations when needed. By combining symbolic reasoning with online motion planning, our system achieves greater flexibility in handling speech corrections and dynamically changing constraints. We evaluate our framework in real-world human-robot interaction scenarios, demonstrating online adaptions of goal poses, constraints, or task objectives. Our results highlight the advantages of integrating incremental language understanding with real-time motion planning for natural and fluid human-robot collaboration. The experiments are demonstrated in the accompanying video at www.acin.tuwien.ac.at/42d5.

8 pages, 9 figures, accepted at IROS 2025

References in corpus (4)

• Text2Motion: From Natural Language Instructions to Feasible Plans
• BoundMPC: Cartesian path following with error bounds based on model predictive control in the joint space
• BoundPlanner: A convex-set-based approach to bounded manipulator trajectory planning
• Incremental processing of noisy user utterances in the spoken language understanding task