Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges

Abstract

Visual saliency models have enjoyed a big leap in performance in recent years, thanks to advances in deep learning and large scale annotated data. Despite enormous effort and huge breakthroughs, however, models still fall short in reaching human-level accuracy. In this work, I explore the landscape of the field emphasizing on new deep saliency models, benchmarks, and datasets. A large number of image and video saliency models are reviewed and compared over two image benchmarks and two large scale video datasets. Further, I identify factors that contribute to the gap between models and humans and discuss remaining issues that need to be addressed to build the next generation of more powerful saliency models. Some specific questions that are addressed include: in what ways current models fail, how to remedy them, what can be learned from cognitive studies of attention, how explicit saliency judgments relate to fixations, how to conduct fair model comparison, and what are the emerging applications of saliency models.

References in corpus (22)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Understanding Neural Networks Through Deep Visualization
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Video Salient Object Detection via Fully Convolutional Networks
• SalGAN: Visual Saliency Prediction with Generative Adversarial Networks
• TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
• CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research
• Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM
• Faster gaze prediction with dense networks and Fisher pruning
• Recurrent Mixture Density Network for Spatiotemporal Visual Attention
• SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes
• Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
• SG-FCN: A Motion and Memory-Based Deep Learning Model for Video Saliency Detection
• Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features
• Understanding Infographics through Textual and Visual Tag Prediction
• Crowdsourcing Gaze Data Collection
• PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks
• Guiding human gaze with convolutional neural networks
• Invariance Analysis of Saliency Models versus Human Gaze During Scene Free Viewing
• The Effect of Distortions on the Prediction of Visual Attention
• Learning a Saliency Evaluation Metric Using Crowdsourced Perceptual Judgments

Cited by in corpus (7)

• Contextual Encoder-Decoder Network for Visual Saliency Prediction
• Salient Objects in Clutter
• Ultrasound Image Representation Learning by Modeling Sonographer Visual Attention
• Where and What: Driver Attention-based Object Detection
• Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts
• ATSal: An Attention Based Architecture for Saliency Prediction in 360 Videos
• State-of-the-art in 360° Video/Image Processing: Perception, Assessment and Compression