Video Storytelling: Textual Summaries for Events
arXiv:1807.09418 · doi:10.1109/TMM.2019.2930041
Abstract
Bridging vision and natural language is a longstanding goal in computer vision and multimedia research. While earlier works focus on generating a single-sentence description for visual content, recent works have studied paragraph generation. In this work, we introduce the problem of video storytelling, which aims at generating coherent and succinct stories for long videos. Video storytelling introduces new challenges, mainly due to the diversity of the story and the length and complexity of the video. We propose novel methods to address the challenges. First, we propose a context-aware framework for multimodal embedding learning, where we design a Residual Bidirectional Recurrent Neural Network to leverage contextual information from past and future. Second, we propose a Narrator model to discover the underlying storyline. The Narrator is formulated as a reinforcement learning agent which is trained by directly optimizing the textual metric of the generated story. We evaluate our method on the Video Story dataset, a new dataset that we have collected to enable the study. We compare our method with multiple state-of-the-art baselines, and show that our method achieves better performance, in terms of quantitative measures and user study.
arXiv commentPublished in IEEE Transactions on Multimedia
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Recurrent Models of Visual Attention
- Sequence Level Training with Recurrent Neural Networks
- Multiple Object Recognition with Visual Attention
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- Coherent Multi-Sentence Video Description with Variable Level of Detail
- Predicting Visual Features from Text for Image and Video Caption Retrieval
- Unsupervised Learning of View-invariant Action Representations
- Recurrent Topic-Transition GAN for Visual Paragraph Generation
- VideoSET: Video Summary Evaluation through Text