SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes

Abstract

We introduce SaltiNet, a deep neural network for scanpath prediction trained on 360-degree images. The model is based on a temporal-aware novel representation of saliency information named the saliency volume. The first part of the network consists of a model trained to generate saliency volumes, whose parameters are fit by back-propagation computed from a binary cross entropy (BCE) loss over downsampled versions of the saliency volumes. Sampling strategies over these volumes are used to generate scanpaths over the 360-degree images. Our experiments show the advantages of using saliency volumes, and how they can be used for related tasks. Our source code and trained models available at https://github.com/massens/saliency-360salient-2017.

arXiv commentWinner of the Best Scan-path Award at the Salient360!: Visual attention modeling for 360 degrees Images Grand Challenge of ICME 2017. Presented at the ICCV 2017 Workshop on Egocentric Perception, Interaction and Computing (EPIC)

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SalGAN: Visual Saliency Prediction with Generative Adversarial Networks
• DeepGaze II: Reading fixations from deep features trained on object recognition
• TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
• A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection
• Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video

Cited by in corpus (13)

• Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach
• ScanGAN360: A Generative Model of Realistic Scanpaths for 360${}^{\circ }$ Images
• Systematic Review of Extended Reality for Smart Built Environments Lighting Design Simulations
• Deep Learning for Content-based Personalized Viewport Prediction of 360-Degree VR Videos
• Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges
• Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts
• A Comparative Study of Scanpath Models in Graph-Based Visualization
• State-of-the-art in 360° Video/Image Processing: Perception, Assessment and Compression
• Wave Propagation of Visual Stimuli in Focus of Attention
• Saliency Prediction on Omnidirectional Images with Generative Adversarial Imitation Learning
• Viewport-Aware Deep Reinforcement Learning Approach for 360${}^o$ Video Caching
• Visual Attention Graph
• Saliency Map Estimation for Omni-Directional Image Considering Prior Distributions