ScanGAN360: A Generative Model of Realistic Scanpaths for 360${}^{\circ }$ Images

Abstract

Understanding and modeling the dynamics of human gaze behavior in 360${}^{\circ }$ environments is a key challenge in computer vision and virtual reality. Generative adversarial approaches could alleviate this challenge by generating a large number of possible scanpaths for unseen images. Existing methods for scanpath generation, however, do not adequately predict realistic scanpaths for 360${}^{\circ }$ images. We present ScanGAN360, a new generative adversarial approach to address this challenging problem. Our network generator is tailored to the specifics of 360${}^{\circ }$ images representing immersive environments. Specifically, we accomplish this by leveraging the use of a spherical adaptation of dynamic-time warping as a loss function and proposing a novel parameterization of 360${}^{\circ }$ scanpaths. The quality of our scanpaths outperforms competing approaches by a large margin and is almost on par with the human baseline. ScanGAN360 thus allows fast simulation of large numbers of virtual observers, whose behavior mimics real users, enabling a better understanding of gaze behavior and novel applications in virtual scene design.

References in corpus (3)

• DeepGaze II: Reading fixations from deep features trained on object recognition
• SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes
• State-of-the-Art in Human Scanpath Prediction

Cited by in corpus (7)

• A Comparative Study of Scanpath Models in Graph-Based Visualization
• Impact of Design Decisions in Scanpath Modeling
• GazeFusion: Saliency-Guided Image Generation
• The Shortest Route Is Not Always the Fastest: Probability-Modeled Stereoscopic Eye Movement Completion Time in VR
• Generating Realistic Synthetic Head Rotation Data for Extended Reality using Deep Learning
• 360° Stereo Image Composition with Depth Adaption
• An Overview on Generative AI at Scale with Edge-Cloud Computing