GazeNoter: Co-Piloted AR Note-Taking via Gaze Selection of LLM Suggestions to Match Users' Intentions

Abstract

Note-taking is critical during speeches and discussions, serving not only for later summarization and organization but also for real-time question and opinion reminding in question-and-answer sessions or timely contributions in discussions. Manually typing on smartphones for note-taking could be distracting and increase cognitive load for users. While large language models (LLMs) are used to automatically generate summaries and highlights, the content generated by artificial intelligence (AI) may not match users' intentions without user input or interaction. Therefore, we propose an AI-copiloted augmented reality (AR) system, GazeNoter, to allow users to swiftly select diverse LLM-generated suggestions via gaze on an AR headset for real-time note-taking. GazeNoter leverages an AR headset as a medium for users to swiftly adjust the LLM output to match their intentions, forming a user-in-the-loop AI system for both within-context and beyond-context notes. We conducted two user studies to verify the usability of GazeNoter in attending speeches in a static sitting condition and walking meetings and discussions in a mobile walking condition, respectively.

22 pages, 19 figures

References in corpus (8)

• Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
• Graphologue: Exploring Large Language Model Responses with Interactive Diagrams
• Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries
• RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling
• ConceptEVA: Concept-Based Interactive Exploration and Customization of Document Summaries
• Gesture-aware Interactive Machine Teaching with In-situ Object Annotations
• picoRing: battery-free rings for subtle thumb-to-index input
• The Walking Talking Stick: Understanding Automated Note-Taking in Walking Meetings