SG-FCN: A Motion and Memory-Based Deep Learning Model for Video Saliency Detection

Abstract

Data-driven saliency detection has attracted strong interest as a result of applying convolutional neural networks to the detection of eye fixations. Although a number of imagebased salient object and fixation detection models have been proposed, video fixation detection still requires more exploration. Different from image analysis, motion and temporal information is a crucial factor affecting human attention when viewing video sequences. Although existing models based on local contrast and low-level features have been extensively researched, they failed to simultaneously consider interframe motion and temporal information across neighboring video frames, leading to unsatisfactory performance when handling complex scenes. To this end, we propose a novel and efficient video eye fixation detection model to improve the saliency detection performance. By simulating the memory mechanism and visual attention mechanism of human beings when watching a video, we propose a step-gained fully convolutional network by combining the memory information on the time axis with the motion information on the space axis while storing the saliency information of the current frame. The model is obtained through hierarchical training, which ensures the accuracy of the detection. Extensive experiments in comparison with 11 state-of-the-art methods are carried out, and the results show that our proposed model outperforms all 11 methods across a number of publicly available datasets.

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Exploring Models and Data for Remote Sensing Image Caption Generation
• DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection
• Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features

Cited by in corpus (9)

• Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges
• Motion Guided Attention for Video Salient Object Detection
• Synthetic Defocus and Look-Ahead Autofocus for Casual Videography
• Visual Sensation and Perception Computational Models for Deep Learning: State of the art, Challenges and Prospects
• A Plug-and-play Scheme to Adapt Image Saliency Deep Model for Video Data
• A Bioinspired Retinal Neural Network for Accurately Extracting Small-Target Motion Information in Cluttered Backgrounds
• Global and Local Sensitivity Guided Key Salient Object Re-augmentation for Video Saliency Detection
• Appearance Fusion of Multiple Cues for Video Co-localization
• Depthwise Non-local Module for Fast Salient Object Detection Using a Single Thread