Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction

Abstract

Most mobile health apps employ data visualization to help people view their health and activity data, but these apps provide limited support for visual data exploration. Furthermore, despite its huge potential benefits, mobile visualization research in the personal data context is sparse. This work aims to empower people to easily navigate and compare their personal health data on smartphones by enabling flexible time manipulation with speech. We designed and developed Data@Hand, a mobile app that leverages the synergy of two complementary modalities: speech and touch. Through an exploratory study with 13 long-term Fitbit users, we examined how multimodal interaction helps participants explore their own health data. Participants successfully adopted multimodal interaction (i.e., speech and touch) for convenient and fluid data exploration. Based on the quantitative and qualitative findings, we discuss design implications and opportunities with multimodal interaction for better supporting visual data exploration on mobile devices.

To appear in ACM CHI 2021 Conference on Human Factors in Computing Systems; 16 pages, 6 figures, 5 tables

References in corpus (4)

• Touch? Speech? or Touch and Speech? Investigating Multimodal Interaction for Visual Network Exploration and Analysis
• Interweaving Multimodal Interaction with Flexible Unit Visualizations for Data Exploration
• Activity River: Visualizing Planned and Logged Personal Activities for Reflection
• A Field Study of On-Calendar Visualizations

Cited by in corpus (7)

• ChaCha: Leveraging Large Language Models to Prompt Children to Share Their Emotions about Personal Events
• AVscript: Accessible Video Editing with Audio-Visual Scripts
• MyMove: Facilitating Older Adults to Collect In-Situ Activity Labels on a Smartwatch with Speech
• ExploreSelf: Fostering User-driven Exploration and Reflection on Personal Challenges with Adaptive Guidance by Large Language Models
• InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions
• "Piecing Data Connections Together Like a Puzzle": Effects of Increasing Task Complexity on the Effectiveness of Data Storytelling Enhanced Visualisations
• Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM