Spatio-Temporal Self-Attention Network for Video Saliency Prediction

Abstract

3D convolutional neural networks have achieved promising results for video tasks in computer vision, including video saliency prediction that is explored in this paper. However, 3D convolution encodes visual representation merely on fixed local spacetime according to its kernel size, while human attention is always attracted by relational visual features at different time. To overcome this limitation, we propose a novel Spatio-Temporal Self-Attention 3D Network (STSANet) for video saliency prediction, in which multiple Spatio-Temporal Self-Attention (STSA) modules are employed at different levels of 3D convolutional backbone to directly capture long-range relations between spatio-temporal features of different time steps. Besides, we propose an Attentional Multi-Scale Fusion (AMSF) module to integrate multi-level features with the perception of context in semantic and spatio-temporal subspaces. Extensive experiments demonstrate the contributions of key components of our method, and the results on DHF1K, Hollywood-2, UCF, and DIEM benchmark datasets clearly prove the superiority of the proposed model compared with all state-of-the-art models.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Video Salient Object Detection via Fully Convolutional Networks
• Deep Visual Attention Prediction
• Dynamic Coattention Networks For Question Answering
• Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model
• Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet
• Contextual Encoder-Decoder Network for Visual Saliency Prediction
• Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM
• Unified Image and Video Saliency Modeling
• How is Gaze Influenced by Image Transformations? Dataset and Model
• Personal Fixations-Based Object Segmentation with Object Localization and Boundary Preservation
• Visual Saliency Prediction Using a Mixture of Deep Neural Networks
• Temporal-Spatial Feature Pyramid for Video Saliency Detection
• Temporal Memory Attention for Video Semantic Segmentation