Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks

Abstract

We introduce Augmented Math, a machine learning-based approach to authoring AR explorable explanations by augmenting static math textbooks without programming. To augment a static document, our system first extracts mathematical formulas and figures from a given document using optical character recognition (OCR) and computer vision. By binding and manipulating these extracted contents, the user can see the interactive animation overlaid onto the document through mobile AR interfaces. This empowers non-technical users, such as teachers or students, to transform existing math textbooks and handouts into on-demand and personalized explorable explanations. To design our system, we first analyzed existing explorable math explanations to identify common design strategies. Based on the findings, we developed a set of augmentation techniques that can be automatically generated based on the extracted content, which are 1) dynamic values, 2) interactive figures, 3) relationship highlights, 4) concrete examples, and 5) step-by-step hints. To evaluate our system, we conduct two user studies: preliminary user testing and expert interviews. The study results confirm that our system allows more engaging experiences for learning math concepts.

arXiv commentUIST 2023

References in corpus (10)

• CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization
• Learnable Programming: Blocks and Beyond
• RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching
• Direct-Manipulation Visualization of Deep Networks
• Augmenting Static Visualizations with PapARVis Designer
• Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching
• Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI
• CrossData: Leveraging Text-Data Connections for Authoring Data Documents
• RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling
• Fidyll: A Compiler for Cross-Format Data Stories & Explorable Explanations

Cited by in corpus (8)

• Augmented Object Intelligence with XR-Objects
• Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams
• Everyday AR through AI-in-the-Loop
• Breaking the Plane: Exploring Real-Time Visualization of 3D Surfaces in Augmented Reality with Handwritten Input
• RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models
• Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations
• Empowering Children to Create AI-Enabled Augmented Reality Experiences
• Evaluating Interactivity: Toward Automated Assessment of AI-Generated Explorable Explanations