VRSight: An AI-Driven Scene Description System to Improve Virtual Reality Accessibility for Blind People

Abstract

Virtual Reality (VR) is inaccessible to blind people. While research has investigated many techniques to enhance VR accessibility, they require additional developer effort to integrate. As such, most mainstream VR apps remain inaccessible as the industry de-prioritizes accessibility. We present VRSight, an end-to-end system that recognizes VR scenes post hoc through a set of AI models (e.g., object detection, depth estimation, LLM-based atmosphere interpretation) and generates tone-based, spatial audio feedback, empowering blind users to interact in VR without developer intervention. To enable virtual element detection, we further contribute DISCOVR, a VR dataset consisting of 30 virtual object classes from 17 social VR apps, substituting real-world datasets that remain not applicable to VR contexts. Nine participants used VRSight to explore an off-the-shelf VR app (Rec Room), demonstrating its effectiveness in facilitating social tasks like avatar awareness and available seat identification.

17 pages, 10 figures, 2 tables, LaTeX; To be published in ACM's 2025 Symposium on User Interface Software and Technology (UIST 2025)

References in corpus (7)

• VRBubble: Enhancing Peripheral Awareness of Avatars for People with Visual Impairments in Social Virtual Reality
• NavStick: Making Video Games Blind-Accessible via the Ability to Look Around
• Making Mobile Augmented Reality Applications Accessible
• Making Short-Form Videos Accessible with Hierarchical Video Summaries
• Uncovering Visually Impaired Gamers' Preferences for Spatial Awareness Tools Within Video Games
• Audio Description Customization
• Accessible Nonverbal Cues to Support Conversations in VR for Blind and Low Vision People