How People Prompt to Create Interactive VR Scenes

Abstract

Generative AI tools can provide people with the ability to create virtual environments and scenes with natural language prompts. Yet, how people will formulate such prompts is unclear -- particularly when they inhabit the environment that they are designing. For instance, it is likely that a person might say, "Put a chair here", while pointing at a location. If such linguistic features are common to people's prompts, we need to tune models to accommodate them. In this work, we present a wizard-of-oz elicitation study with 22 participants, where we studied people's implicit expectations when verbally prompting such programming agents to create interactive VR scenes. Our findings show that people prompt with several implicit expectations: (1) that agents have an embodied knowledge of the environment; (2) that agents understand embodied prompts by users; (3) that the agents can recall previous states of the scene and the conversation, and that (4) agents have a commonsense understanding of objects in the scene. Further, we found that participants prompt differently when they are prompting in situ (i.e. within the VR environment) versus ex situ (i.e. viewing the VR environment from the outside). To explore how our could be applied, we designed and built Oastaad, a conversational programming agent that allows non-programmers to design interactive VR experiences that they inhabit. Based on these explorations, we outline new opportunities and challenges for conversational programming agents that create VR environments.

Accepted at ACM 2024 Designing Interactive Systems (DIS)

References in corpus (9)

• Evaluating Large Language Models Trained on Code
• ReAct: Synergizing Reasoning and Acting in Language Models
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models
• Eliciting and Analysing Users' Envisioned Dialogues with Perfect Voice Assistants
• A Wizard of Oz Study Simulating API Usage Dialogues with a Virtual Assistant
• LLMR: Real-time Prompting of Interactive Worlds using Large Language Models
• ProAgent: Building Proactive Cooperative Agents with Large Language Models
• Canvil: Designerly Adaptation for LLM-Powered User Experiences

Cited by in corpus (6)

• ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality
• Everyday AR through AI-in-the-Loop
• Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes
• Memory Remedy: An AI-Enhanced Interactive Story Exploring Human-Robot Interaction and Companionship
• HandProxy: Expanding the Affordances of Speech Interfaces in Immersive Environments with a Virtual Proxy Hand
• Empowering Children to Create AI-Enabled Augmented Reality Experiences