SSAN: Separable Self-Attention Network for Video Representation Learning

Abstract

Self-attention has been successfully applied to video representation learning due to the effectiveness of modeling long range dependencies. Existing approaches build the dependencies merely by computing the pairwise correlations along spatial and temporal dimensions simultaneously. However, spatial correlations and temporal correlations represent different contextual information of scenes and temporal reasoning. Intuitively, learning spatial contextual information first will benefit temporal modeling. In this paper, we propose a separable self-attention (SSA) module, which models spatial and temporal correlations sequentially, so that spatial contexts can be efficiently used in temporal modeling. By adding SSA module into 2D CNN, we build a SSA network (SSAN) for video representation learning. On the task of video action recognition, our approach outperforms state-of-the-art methods on Something-Something and Kinetics-400 datasets. Our models often outperform counterparts with shallower network and fewer modalities. We further verify the semantic learning ability of our method in visual-language task of video retrieval, which showcases the homogeneity of video representations and text embeddings. On MSR-VTT and Youcook2 datasets, video representations learnt by SSA significantly improve the state-of-the-art performance.

arXiv commentAccepted by CVPR2021

References in corpus (9)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Spatiotemporal Residual Networks for Video Action Recognition
• Aggregated Residual Transformations for Deep Neural Networks
• Learning Video Representations using Contrastive Bidirectional Transformer
• More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language
• End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering