Reconstructive Sequence-Graph Network for Video Summarization

Abstract

Exploiting the inner-shot and inter-shot dependencies is essential for key-shot based video summarization. Current approaches mainly devote to modeling the video as a frame sequence by recurrent neural networks. However, one potential limitation of the sequence models is that they focus on capturing local neighborhood dependencies while the high-order dependencies in long distance are not fully exploited. In general, the frames in each shot record a certain activity and vary smoothly over time, but the multi-hop relationships occur frequently among shots. In this case, both the local and global dependencies are important for understanding the video content. Motivated by this point, we propose a Reconstructive Sequence-Graph Network (RSGN) to encode the frames and shots as sequence and graph hierarchically, where the frame-level dependencies are encoded by Long Short-Term Memory (LSTM), and the shot-level dependencies are captured by the Graph Convolutional Network (GCN). Then, the videos are summarized by exploiting both the local and global dependencies among shots. Besides, a reconstructor is developed to reward the summary generator, so that the generator can be optimized in an unsupervised manner, which can avert the lack of annotated data in video summarization. Furthermore, under the guidance of reconstruction loss, the predicted summary can better preserve the main video content and shot-level dependencies. Practically, the experimental results on three popular datasets i.e., SumMe, TVsum and VTW) have demonstrated the superiority of our proposed approach to the summarization task.

arXiv commentAccepted by IEEE TPAMI 2021

References in corpus (10)

• Semi-Supervised Classification with Graph Convolutional Networks
• Structured Neural Summarization
• SocialGCN: An Efficient Graph Convolutional Network based Model for Social Recommendation
• Video Summarization with Long Short-term Memory
• Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward
• Convolutional Hierarchical Attention Network for Query-Focused Video Summarization
• Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network
• Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach
• Attention is all you need for Videos: Self-attention based Video Summarization using Universal Transformers
• Title Generation for User Generated Videos

Cited by in corpus (4)

• MHSA-Net: Multi-Head Self-Attention Network for Occluded Person Re-Identification
• Video Joint Modelling Based on Hierarchical Transformer for Co-summarization
• Low-Light Hyperspectral Image Enhancement
• Audio-visual Representation Learning for Anomaly Events Detection in Crowds