From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality

Abstract

We introduce Speech-to-Spatial, a referent disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance. Unlike prior systems that rely on additional cues (e.g., gesture, gaze) or manual expert annotations, Speech-to-Spatial infers the intended target solely from spoken references (speech input). Motivated by our formative study of speech referencing patterns, we characterize recurring ways people specify targets (Direct Attribute, Relational, Remembrance, and Chained) and ground them to our object-centric relational graph. Given an utterance, referent cues are parsed and rendered as persistent in-situ AR visual guidance, reducing iterative micro-guidance ("a bit more to the right", "now, stop.") during remote guidance. We demonstrate the use cases of our system with remote guided assistance and intent disambiguation scenarios. Our evaluation shows that Speechto-Spatial improves task efficiency, reduces cognitive load, and enhances usability compared to a conventional voice-only baseline, transforming disembodied verbal instruction into visually explainable, actionable guidance on a live shared view.

11 pages, 6 figures. This is the author's version of the article that appeared at the IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR) 2026

References in corpus (12)

• GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality
• Design Patterns for Situated Visualization in Augmented Reality
• Augmented Object Intelligence with XR-Objects
• Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory Augmentation
• Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework
• Learning to Generate Unambiguous Spatial Referring Expressions for Real-World Environments
• ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality
• Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agents
• Remote Assistance with Mixed Reality for Procedural Tasks
• Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models
• Interactive and Incremental Learning of Spatial Object Relations from Human Demonstrations
• Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations