Contextual Encoder-Decoder Network for Visual Saliency Prediction

Abstract

Predicting salient regions in natural images requires the detection of objects that are present in a scene. To develop robust representations for this challenging task, high-level visual features at multiple spatial scales must be extracted and augmented with contextual information. However, existing models aimed at explaining human fixation maps do not incorporate such a mechanism explicitly. Here we propose an approach based on a convolutional neural network pre-trained on a large-scale image classification task. The architecture forms an encoder-decoder structure and includes a module with multiple convolutional layers at different dilation rates to capture multi-scale features in parallel. Moreover, we combine the resulting representations with global scene information for accurately predicting visual saliency. Our model achieves competitive and consistent results across multiple evaluation metrics on two public saliency benchmarks and we demonstrate the effectiveness of the suggested approach on five datasets and selected examples. Compared to state of the art approaches, the network is based on a lightweight image classification backbone and hence presents a suitable choice for applications with limited computational resources, such as (virtual) robotic systems, to estimate human fixations across complex natural scenes.

Updated contact information

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Rethinking Atrous Convolution for Semantic Image Segmentation
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Fully Convolutional Networks for Semantic Segmentation
• Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model
• SalGAN: Visual Saliency Prediction with Generative Adversarial Networks
• DeepGaze II: Reading fixations from deep features trained on object recognition
• TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
• CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research
• Dual Path Networks
• Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges
• Emergence of foveal image sampling from learning to attend in visual scenes
• Invariance Analysis of Saliency Models versus Human Gaze During Scene Free Viewing
• Influence of Image Classification Accuracy on Saliency Map Estimation

Cited by in corpus (11)

• TranSalNet: Towards perceptually relevant visual saliency prediction
• Spatio-Temporal Self-Attention Network for Video Saliency Prediction
• Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency
• SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation
• Fast Hybrid Image Retargeting
• Emergence of Human-Like Attention in Self-Supervised Vision Transformers: an eye-tracking study
• Wave Propagation of Visual Stimuli in Focus of Attention
• Instance-Level Relative Saliency Ranking with Graph Reasoning
• Deriving Explanation of Deep Visual Saliency Models
• Sparsity-Control Ternary Weight Networks
• Saliency Prediction with External Knowledge