Saliency Prediction with External Knowledge

Abstract

The last decades have seen great progress in saliency prediction, with the success of deep neural networks that are able to encode high-level semantics. Yet, while humans have the innate capability in leveraging their knowledge to decide where to look (e.g. people pay more attention to familiar faces such as celebrities), saliency prediction models have only been trained with large eye-tracking datasets. This work proposes to bridge this gap by explicitly incorporating external knowledge for saliency models as humans do. We develop networks that learn to highlight regions by incorporating prior knowledge of semantic relationships, be it general or domain-specific, depending on the task of interest. At the core of the method is a new Graph Semantic Saliency Network (GraSSNet) that constructs a graph that encodes semantic relationships learned from external knowledge. A Spatial Graph Attention Network is then developed to update saliency features based on the learned graph. Experiments show that the proposed model learns to predict saliency from the external knowledge and outperforms the state-of-the-art on four saliency benchmarks.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Spectral Networks and Locally Connected Networks on Graphs
• Deep Convolutional Networks on Graph-Structured Data
• Verb Semantics and Lexical Selection
• Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet
• Contextual Encoder-Decoder Network for Visual Saliency Prediction
• Shallow and Deep Convolutional Networks for Saliency Prediction
• Hybrid Knowledge Routed Modules for Large-scale Object Detection
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• A Dilated Inception Network for Visual Saliency Prediction