DeepGaze II: Reading fixations from deep features trained on object recognition

Abstract

Here we present DeepGaze II, a model that predicts where people look in images. The model uses the features from the VGG-19 deep neural network trained to identify objects in images. Contrary to other saliency models that use deep features, here we use the VGG features for saliency prediction with no additional fine-tuning (rather, a few readout layers are trained on top of the VGG features to predict saliency). The model is therefore a strong test of transfer learning. After conservative cross-validation, DeepGaze II explains about 87% of the explainable information gain in the patterns of fixations and achieves top performance in area under the curve metrics on the MIT300 hold-out benchmark. These results corroborate the finding from DeepGaze I (which explained 56% of the explainable information gain), that deep features trained on object recognition provide a versatile feature space for performing related visual tasks. We explore the factors that contribute to this success and present several informative image examples. A web service is available to compute model predictions at http://deepgaze.bethgelab.org.

Cited by in corpus (35)

• Generalisation in humans and deep neural networks
• Contextual Encoder-Decoder Network for Visual Saliency Prediction
• SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes
• ScanGAN360: A Generative Model of Realistic Scanpaths for 360${}^{\circ }$ Images
• DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction
• Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency
• Attention Flow: End-to-End Joint Attention Estimation
• Human Attention in Fine-grained Classification
• Where and What: Driver Attention-based Object Detection
• A Mathematical Model of Local and Global Attention in Natural Scene Viewing
• A Dilated Inception Network for Visual Saliency Prediction
• See, Attend and Brake: An Attention-based Saliency Map Prediction Model for End-to-End Driving
• Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention
• DeepFeat: A Bottom Up and Top Down Saliency Model Based on Deep Features of Convolutional Neural Nets
• Salient Region Segmentation
• Saliency for free: Saliency prediction as a side-effect of object recognition
• Saccade Sequence Prediction: Beyond Static Saliency Maps
• Semantic and Contrast-Aware Saliency
• Predicting Driver Attention in Critical Situations
• Wave Propagation of Visual Stimuli in Focus of Attention
• Deep Learning--Based Scene Simplification for Bionic Vision
• Data augmentation and image understanding
• Rapid Visual Categorization is not Guided by Early Salience-Based Selection
• Influence of Image Classification Accuracy on Saliency Map Estimation
• Saliency Preservation in Low-Resolution Grayscale Images
• SalGaze: Personalizing Gaze Estimation Using Visual Saliency
• Disentangling top-down vs. bottom-up and low-level vs. high-level influences on eye movements over time
• Light-weighted Saliency Detection with Distinctively Lower Memory Cost and Model Size
• Implicit Saliency in Deep Neural Networks
• Saliency Map Estimation for Omni-Directional Image Considering Prior Distributions
• Deep Learning Approach Protecting Privacy in Camera-Based Critical Applications
• Ultrafast Video Attention Prediction with Coupled Knowledge Distillation
• Learning Aesthetic Layouts via Visual Guidance
• FastSal: a Computationally Efficient Network for Visual Saliency Prediction
• Grounded Human-Object Interaction Hotspots from Video (Extended Abstract)