VRCopilot: Authoring 3D Layouts with Generative AI Models in VR

Abstract

Immersive authoring provides an intuitive medium for users to create 3D scenes via direct manipulation in Virtual Reality (VR). Recent advances in generative AI have enabled the automatic creation of realistic 3D layouts. However, it is unclear how capabilities of generative AI can be used in immersive authoring to support fluid interactions, user agency, and creativity. We introduce VRCopilot, a mixed-initiative system that integrates pre-trained generative AI models into immersive authoring to facilitate human-AI co-creation in VR. VRCopilot presents multimodal interactions to support rapid prototyping and iterations with AI, and intermediate representations such as wireframes to augment user controllability over the created content. Through a series of user studies, we evaluated the potential and challenges in manual, scaffolded, and automatic creation in immersive authoring. We found that scaffolded creation using wireframes enhanced the user agency compared to automatic creation. We also found that manual creation via multimodal specification offers the highest sense of creativity and agency.

arXiv commentUIST 2024

References in corpus (6)

• GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
• How the Design of YouTube Influences User Sense of Agency
• Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators
• Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints
• GRAINS: Generative Recursive Autoencoders for INdoor Scenes
• ATISS: Autoregressive Transformers for Indoor Scene Synthesis

Cited by in corpus (6)

• ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality
• InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
• EchoLadder: Progressive AI-Assisted Design of Immersive VR Scenes
• HandProxy: Expanding the Affordances of Speech Interfaces in Immersive Environments with a Virtual Proxy Hand
• Empowering Children to Create AI-Enabled Augmented Reality Experiences
• SonoCraftAR: Towards Supporting Personalized Authoring of Sound-Reactive AR Interfaces by Deaf and Hard of Hearing Users