Shallow and Deep Convolutional Networks for Saliency Prediction

Abstract

The prediction of salient areas in images has been traditionally addressed with hand-crafted features based on neuroscience principles. This paper, however, addresses the problem with a completely data-driven approach by training a convolutional neural network (convnet). The learning process is formulated as a minimization of a loss function that measures the Euclidean distance of the predicted saliency map with the provided ground truth. The recent publication of large datasets of saliency prediction has provided enough data to train end-to-end architectures that are both fast and accurate. Two designs are proposed: a shallow convnet trained from scratch, and a another deeper solution whose first three layers are adapted from another network trained for classification. To the authors knowledge, these are the first end-to-end CNNs trained and tested for the purpose of saliency prediction.

Preprint of the paper accepted at 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Source code and models available at https://github.com/imatge-upc/saliency-2016-cvpr. Junting Pan and Kevin McGuinness contributed equally to this work

References in corpus (8)

• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Theano: new features and speed improvements
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
• Visual Saliency Based on Multiscale Deep Features
• Pixels to Voxels: Modeling Visual Representation in the Human Brain

Cited by in corpus (40)

• CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes
• Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features
• Revisiting Video Saliency: A Large-scale Benchmark and a New Model
• Understanding Infographics through Textual and Visual Tag Prediction
• Co-salient Object Detection Based on Deep Saliency Networks and Seed Propagation over an Integrated Graph
• Bio-Inspired Representation Learning for Visual Attention Prediction
• Simple vs complex temporal recurrences for video saliency prediction
• TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection
• How do people explore virtual environments?
• Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video
• Context Proposals for Saliency Detection
• Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition
• A Dilated Inception Network for Visual Saliency Prediction
• Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention
• The Devil is in the Decoder: Classification, Regression and GANs
• Unsupervised Self-training Algorithm Based on Deep Learning for Optical Aerial Images Change Detection
• Goal-Oriented Gaze Estimation for Zero-Shot Learning
• Saliency for free: Saliency prediction as a side-effect of object recognition
• State-of-the-art in 360° Video/Image Processing: Perception, Assessment and Compression
• Image Cropping with Composition and Saliency Aware Aesthetic Score Map
• Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes
• An End-to-End Neural Network for Image Cropping by Learning Composition from Aesthetic Photos
• SalSum: Saliency-based Video Summarization using Generative Adversarial Networks
• Enhancing Salient Object Segmentation Through Attention
• Cube Padding for Weakly-Supervised Saliency Prediction in 360° Videos
• Saliency Revisited: Analysis of Mouse Movements versus Fixations
• Webpage Saliency Prediction with Two-stage Generative Adversarial Networks
• A-Lamp: Adaptive Layout-Aware Multi-Patch Deep Convolutional Neural Network for Photo Aesthetic Assessment
• Saliency Prediction on Omnidirectional Images with Generative Adversarial Imitation Learning
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• A Self Validation Network for Object-Level Human Attention Estimation
• Detecting Attended Visual Targets in Video
• Saliency Prediction with External Knowledge
• Compare and Contrast: Learning Prominent Visual Differences
• FastSal: a Computationally Efficient Network for Visual Saliency Prediction
• Visual Attention on the Sun: What Do Existing Models Actually Predict?
• SUSiNet: See, Understand and Summarize it
• Implicit Saliency in Deep Neural Networks
• Salient Bundle Adjustment for Visual SLAM
• Visual Attention for Behavioral Cloning in Autonomous Driving