Empowering Children to Create AI-Enabled Augmented Reality Experiences

Abstract

Despite their potential to enhance children's learning experiences, AI-enabled AR technologies are predominantly used in ways that position children as consumers rather than creators. We introduce Capybara, an AR-based and AI-powered visual programming environment that empowers children to create, customize, and program 3D characters overlaid onto the physical world. Capybara enables children to create virtual characters and accessories using text-to-3D generative AI models, and to animate these characters through auto-rigging and body tracking. In addition, our system employs vision-based AI models to recognize physical objects, allowing children to program interactive behaviors between virtual characters and their physical surroundings. We demonstrate the expressiveness of Capybara through a set of novel AR experiences. We conducted user studies with 20 children in the United States and Argentina. Our findings suggest that Capybara can empower children to harness AI in authoring personalized and engaging AR experiences that seamlessly bridge the virtual and physical worlds.

Accepted to ACM UIST 2025

References in corpus (16)

• Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works
• 3D Gaussian as a New Era: A Survey
• StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement
• PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
• Remixing as a Pathway to Computational Thinking
• RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching
• Mathemyths: Leveraging Large Language Models to Teach Mathematical Language through Child-AI Co-Creative Storytelling
• Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching
• ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12
• VRCopilot: Authoring 3D Layouts with Generative AI Models in VR
• RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling
• Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks
• How People Prompt to Create Interactive VR Scenes
• Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams
• Project IRL: Playful Co-Located Interactions with Mobile Augmented Reality
• From Following to Understanding: Investigating the Role of Reflective Prompts in AR-Guided Tasks to Promote Task Understanding