When Refreshable Tactile Displays Meet Conversational Agents: Investigating Accessible Data Presentation and Analysis with Touch and Speech

Abstract

Despite the recent surge of research efforts to make data visualizations accessible to people who are blind or have low vision (BLV), how to support BLV people's data analysis remains an important and challenging question. As refreshable tactile displays (RTDs) become cheaper and conversational agents continue to improve, their combination provides a promising approach to support BLV people's interactive data exploration and analysis. To understand how BLV people would use and react to a system combining an RTD with a conversational agent, we conducted a Wizard-of-Oz study with 11 BLV participants, where they interacted with line charts, bar charts, and isarithmic maps. Our analysis of participants' interactions led to the identification of nine distinct patterns. We also learned that the choice of modalities depended on the type of task and prior experience with tactile graphics, and that participants strongly preferred the combination of RTD and speech to a single modality. In addition, participants with more tactile experience described how tactile images facilitated a deeper engagement with the data and supported independent interpretation. Our findings will inform the design of interfaces for such interactive mixed-modality systems.

Accepted to be presented at IEEE VIS 2024 (Honorable Mention Award) and published in IEEE TVCG; Replacement: typos corrected, external DOI added

References in corpus (7)

• Technology Developments in Touch-Based Accessible Graphics: A Systematic Review of Research 2010-2020
• SeeChart: Enabling Accessible Visualizations Through Interactive Natural Language Interface For People with Visual Impairments
• MAIDR: Making Statistical Visualizations Accessible with Multimodal Data Representation
• Data Navigator: An accessibility-centered data navigation toolkit
• ChartA11y: Designing Accessible Touch Experiences of Visualizations with Blind Smartphone Users
• "Hey Model!" - Natural User Interactions and Agency in Accessible Interactive 3D Models
• Designing Conversational Multimodal 3D Printed Models with People who are Blind

Cited by in corpus (4)

• StreetReaderAI: Making Street View Accessible Using Context-Aware Multimodal AI
• MapIO: Embodied Interaction for the Accessibility of Tactile Maps Through Augmented Touch Exploration and Conversation
• Three Modalities, Two Design Probes, One Prototype, and No Vision: Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool
• Supporting Multimodal Data Interaction on Refreshable Tactile Displays: An Architecture to Combine Touch and Conversational AI