PaperToPlace: Transforming Instruction Documents into Spatialized and Context-Aware Mixed Reality Experiences

Abstract

While paper instructions are one of the mainstream medium for sharing knowledge, consuming such instructions and translating them into activities are inefficient due to the lack of connectivity with physical environment. We present PaperToPlace, a novel workflow comprising an authoring pipeline, which allows the authors to rapidly transform and spatialize existing paper instructions into MR experience, and a consumption pipeline, which computationally place each instruction step at an optimal location that is easy to read and do not occlude key interaction areas. Our evaluations of the authoring pipeline with 12 participants demonstrated the usability of our workflow and the effectiveness of using a machine learning based approach to help extracting the spatial locations associated with each steps. A second within-subject study with another 12 participants demonstrates the merits of our consumption pipeline by reducing efforts of context switching, delivering the segmented instruction steps and offering the hands-free affordances.

21 pages, 23 figures, Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23), San Francisco, CA, USA

References in corpus (2)

• Investigating Input Modality and Task Geometry on Precision-first 3D Drawing in Virtual Reality
• VRContour: Bringing Contour Delineations of Medical Structures Into Virtual Reality

Cited by in corpus (4)

• Vision-Based Multimodal Interfaces: A Survey and Taxonomy for Enhanced Context-Aware System Design
• Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models
• Augmented Reality Productivity In-the-Wild: A Diary Study of Usage Patterns and Experiences of Working With AR Laptops in Real-World Settings
• Exploring Needs and Design Opportunities for Proactive Information Support in In-Person Small-Group Conversations