Streamlined Dense Video Captioning
arXiv:1904.03870
Abstract
Dense video captioning is an extremely challenging task since accurate and coherent description of events in a video requires holistic understanding of video contents as well as contextual reasoning of individual events. Most existing approaches handle this problem by first detecting event proposals from a video and then captioning on a subset of the proposals. As a result, the generated sentences are prone to be redundant or inconsistent since they fail to consider temporal dependency between events. To tackle this challenge, we propose a novel dense video captioning framework, which models temporal dependency across events in a video explicitly and leverages visual and linguistic context from prior events for coherent storytelling. This objective is achieved by 1) integrating an event sequence generation network to select a sequence of event proposals adaptively, and 2) feeding the sequence of event proposals to our sequential video captioning network, which is trained by reinforcement learning with two-level rewards at both event and episode levels for better context modeling. The proposed technique achieves outstanding performances on ActivityNet Captions dataset in most metrics.
CVPR 2019
References in corpus (10)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Deep Reinforcement Learning-based Image Captioning with Embedding Reward
- Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
- The ActivityNet Large-Scale Activity Recognition Challenge 2018 Summary
- TGIF: A New Dataset and Benchmark on Animated GIF Description
- End-to-End Dense Video Captioning with Masked Transformer
- Reconstruction Network for Video Captioning
- Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning
- Jointly Localizing and Describing Events for Dense Video Captioning