PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks
arXiv:1809.00567
Abstract
We introduce PathGAN, a deep neural network for visual scanpath prediction trained on adversarial examples. A visual scanpath is defined as the sequence of fixation points over an image defined by a human observer with its gaze. PathGAN is composed of two parts, the generator and the discriminator. Both parts extract features from images using off-the-shelf networks, and train recurrent layers to generate or discriminate scanpaths accordingly. In scanpath prediction, the stochastic nature of the data makes it very difficult to generate realistic predictions using supervised learning strategies, but we adopt adversarial training as a suitable alternative. Our experiments prove how PathGAN improves the state of the art of visual scanpath prediction on the iSUN and Salient360! datasets. Source code and models are available at https://imatge-upc.github.io/pathgan/
arXiv commentECCV 2018 Workshop on Egocentric Perception, Interaction and Computing (EPIC). This work obtained the 2nd award in Prediction of Head-gaze Scan-paths for Images, and the 2nd award in Prediction of Eye-gaze Scan-paths for Images at the IEEE ICME 2018 Salient360! Challenge
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Conditional Generative Adversarial Nets
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Generative Adversarial Text to Image Synthesis
- Energy-based Generative Adversarial Network
- TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
- Visual Saliency Based on Multiscale Deep Features
- Eye Tracking for Everyone
Cited by in corpus (4)
- A privacy-preserving approach to streaming eye-tracking data
- Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges
- ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery
- Generating Memorable Images Based on Human Visual Memory Schemas