Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet

Abstract

Recent results suggest that state-of-the-art saliency models perform far from optimal in predicting fixations. This lack in performance has been attributed to an inability to model the influence of high-level image features such as objects. Recent seminal advances in applying deep neural networks to tasks like object recognition suggests that they are able to capture this kind of structure. However, the enormous amount of training data necessary to train these networks makes them difficult to apply directly to saliency prediction. We present a novel way of reusing existing neural networks that have been pretrained on the task of object recognition in models of fixation prediction. Using the well-known network of Krizhevsky et al. (2012), we come up with a new saliency model that significantly outperforms all state-of-the-art models on the MIT Saliency Benchmark. We show that the structure of this network allows new insights in the psychophysics of fixation selection and potentially their neural implementation. To train our network, we build on recent work on the modeling of saliency as point processes.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Going Deeper with Convolutions
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Visualizing and Understanding Convolutional Networks
• Learning Generative Models with Visual Attention

Cited by in corpus (28)

• Deep Visual Attention Prediction
• DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection
• Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model
• Learning Visual Importance for Graphic Designs and Data Visualizations
• DISC: Deep Image Saliency Computing via Progressive Representation Learning
• Object Detection with Deep Learning: A Review
• Spatio-Temporal Self-Attention Network for Video Saliency Prediction
• Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features
• Restricting the Flow: Information Bottlenecks for Attribution
• Visual Saliency Prediction Using a Mixture of Deep Neural Networks
• Bio-Inspired Representation Learning for Visual Attention Prediction
• Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency
• A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection
• A Mathematical Model of Local and Global Attention in Natural Scene Viewing
• A Dilated Inception Network for Visual Saliency Prediction
• DeepFeat: A Bottom Up and Top Down Saliency Model Based on Deep Features of Convolutional Neural Nets
• Improving Video Compression With Deep Visual-Attention Models
• Salient Region Segmentation
• Semantic and Contrast-Aware Saliency
• What Are You Looking at? Detecting Human Intention in Gaze based Human-Robot Interaction
• SalSum: Saliency-based Video Summarization using Generative Adversarial Networks
• Personalized Saliency and its Prediction
• Few-Shot Abstract Visual Reasoning With Spectral Features
• SalGaze: Personalizing Gaze Estimation Using Visual Saliency
• Believe It or Not, We Know What You Are Looking at!
• Visual Attention for Behavioral Cloning in Autonomous Driving
• Saliency Prediction with External Knowledge
• Visual Attention on the Sun: What Do Existing Models Actually Predict?