Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes

Abstract

As more applications of large language models (LLMs) for 3D content for immersive environments emerge, it is crucial to study user behaviour to identify interaction patterns and potential barriers to guide the future design of immersive content creation and editing systems which involve LLMs. In an empirical user study with 12 participants, we combine quantitative usage data with post-experience questionnaire feedback to reveal common interaction patterns and key barriers in LLM-assisted 3D scene editing systems. We identify opportunities for improving natural language interfaces in 3D design tools and propose design recommendations for future LLM-integrated 3D content creation systems. Through an empirical study, we demonstrate that LLM-assisted interactive systems can be used productively in immersive environments.

Published in the IEEE VR (IEEE Conference on Virtual Reality and 3D User Interfaces) 2025 Proceedings

References in corpus (12)

• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• Inner Monologue: Embodied Reasoning through Planning with Language Models
• NExT-GPT: Any-to-Any Multimodal LLM
• GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality
• Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacy
• How People Prompt to Create Interactive VR Scenes
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey
• Eliciting Multimodal Gesture+Speech Interactions in a Multi-Object Augmented Reality Environment
• When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
• Steps towards prompt-based creation of virtual worlds
• VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications