StreetReaderAI: Making Street View Accessible Using Context-Aware Multimodal AI
arXiv:2508.08524 · doi:10.1145/3746059.3747756
Abstract
Interactive streetscape mapping tools such as Google Street View (GSV) and Meta Mapillary enable users to virtually navigate and experience real-world environments via immersive 360° imagery but remain fundamentally inaccessible to blind users. We introduce StreetReaderAI, the first-ever accessible street view tool, which combines context-aware, multimodal AI, accessible navigation controls, and conversational speech. With StreetReaderAI, blind users can virtually examine destinations, engage in open-world exploration, or virtually tour any of the over 220 billion images and 100+ countries where GSV is deployed. We iteratively designed StreetReaderAI with a mixed-visual ability team and performed an evaluation with eleven blind users. Our findings demonstrate the value of an accessible street view in supporting POI investigations and remote route planning. We close by enumerating key guidelines for future work.
Accepted to UIST'25; v2. Fixed a missing word in the PDF; v3. Fixed a typo in an author's name; v4. Changed system name and title
References in corpus (12)
- 3D Printed Maps and Icons for Inclusion: Testing in the Wild by People who are Blind or have Low Vision
- WorldScribe: Towards Context-Aware Live Visual Descriptions
- CitySurfaces: City-Scale Semantic Segmentation of Sidewalk Materials
- Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People
- CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision
- When Refreshable Tactile Displays Meet Conversational Agents: Investigating Accessible Data Presentation and Analysis with Touch and Speech
- Context-Aware Image Descriptions for Web Accessibility
- TactIcons: Designing 3D Printed Map Icons for People who are Blind or have Low Vision
- Engaging with Children's Artwork in Mixed Visual-Ability Families
- An AI Guide to Enhance Accessibility of Social Virtual Reality for Blind People
- ArtInsight: Enabling AI-Powered Artwork Engagement for Mixed Visual-Ability Families
- AltGeoViz: Facilitating Accessible Geovisualization