Gaze-controlled segmentation improves object recognition accuracy in simulated phosphene vision

Abstract

Visual neuroprosthetics provide a simplified visual sense through cameras, computers, and implanted electrodes. A central challenge is deciding what to convey with the few phosphenes available. We asked whether letting the user's gaze select the object that is conveyed in full, rather than extracting edges across the whole scene, improves object recognition in simulated phosphene vision. We developed Gaze-Controlled Segmentation (GCS), in which the point of gaze prompts the Segment Anything Model and the selected object is rendered as a filled region over the scene's edges at reduced intensity. In simulated phosphene vision, eight participants performed an object search-and-localization task under GCS, edge detection, and normal-vision conditions. Both phosphene conditions were gaze-contingent, and data were analysed with Bayesian mixed-effects models. Relative to edge detection, GCS produced a modest but credible improvement in identification accuracy on target-present trials (+9.4 percentage points, 95% highest-density interval [+4.3, +14.2]). Gaze reached the target about as often under both methods, consistent with an advantage at the identification stage. GCS trials tended to be slower (+2.7 s, [-0.8, +6.6]), a difference that was not credible, and in exploratory analyses the advantage was largest for cylindrical objects and absent for round ones. Stated preferences leaned toward GCS on most usability criteria, though with eight participants these are individually consistent with chance. The comparison does not isolate the contribution of the segmentation itself. These findings illustrate the potential of gaze-driven, interactive scene representation for neuroprosthetic vision, while underscoring that the gain is modest, task-dependent, and established so far only in simulated phosphene vision with sighted observers, using phosphenes larger than current implants would produce.