Jointly Localizing and Describing Events for Dense Video Captioning
arXiv:1804.08274
Abstract
Automatically describing a video with natural language is regarded as a fundamental challenge in computer vision. The problem nevertheless is not trivial especially when a video contains multiple events to be worthy of mention, which often happens in real videos. A valid question is how to temporally localize and then describe events, which is known as "dense video captioning." In this paper, we present a novel framework for dense video captioning that unifies the localization of temporal event proposals and sentence generation of each proposal, by jointly training them in an end-to-end manner. To combine these two worlds, we integrate a new design, namely descriptiveness regression, into a single shot detection structure to infer the descriptive complexity of each detected proposal via sentence generation. This in turn adjusts the temporal locations of each event proposal. Our model differs from existing dense video captioning methods since we propose a joint and global optimization of detection and captioning, and the framework uniquely capitalizes on an attribute-augmented video captioning architecture. Extensive experiments are conducted on ActivityNet Captions dataset and our framework shows clear improvements when compared to the state-of-the-art techniques. More remarkably, we obtain a new record: METEOR of 12.96% on ActivityNet Captions official test set.
arXiv commentCVPR 2018 Spotlight, Rank 1 in ActivityNet Captions Challenge 2017
References in corpus (9)
- Sequence to Sequence Learning with Neural Networks
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
- Sequence to Sequence -- Video to Text
- Describing Videos by Exploiting Temporal Structure
- Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
- TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
- ActivityNet Challenge 2017 Summary
- Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
Cited by in corpus (15)
- Exploring Visual Relationship for Image Captioning
- Support-set bottlenecks for video-text representation learning
- Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training
- Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
- YH Technologies at ActivityNet Challenge 2018
- Trimmed Action Recognition, Dense-Captioning Events in Videos, and Spatio-temporal Action Localization with Focus on ActivityNet Challenge 2019
- Mocycle-GAN: Unpaired Video-to-Video Translation
- Streamlined Dense Video Captioning
- Pointing Novel Objects in Image Captioning
- CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
- Grounded Video Description
- X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics
- Multi-modal Dense Video Captioning
- Zero-Shot Activity Recognition with Videos
- Towards Diverse Paragraph Captioning for Untrimmed Videos