RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling

Abstract

We present RealityTalk, a system that augments real-time live presentations with speech-driven interactive virtual elements. Augmented presentations leverage embedded visuals and animation for engaging and expressive storytelling. However, existing tools for live presentations often lack interactivity and improvisation, while creating such effects in video editing tools require significant time and expertise. RealityTalk enables users to create live augmented presentations with real-time speech-driven interactions. The user can interactively prompt, move, and manipulate graphical elements through real-time speech and supporting modalities. Based on our analysis of 177 existing video-edited augmented presentations, we propose a novel set of interaction techniques and then incorporated them into RealityTalk. We evaluate our tool from a presenter's perspective to demonstrate the effectiveness of our system.

UIST 2022; For the interactive gallery, see https://ilab.ucalgary.ca/realitytalk/

References in corpus (2)

• RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching
• HoloBoard: a Large-format Immersive Teaching Board based on pseudo HoloGraphics

Cited by in corpus (6)

• Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching
• Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks
• ChameleonControl: Teleoperating Real Human Surrogates through Mixed Reality Gestural Guidance for Remote Hands-on Classrooms
• ExpressEdit: Video Editing with Natural Language and Sketching
• RealityCanvas: Augmented Reality Sketching for Embedded and Responsive Scribble Animation Effects
• Stargazer: An Interactive Camera Robot for Capturing How-To Videos Based on Subtle Instructor Cues