Weakly Supervised Dense Video Captioning
arXiv:1704.01502
Abstract
This paper focuses on a novel and challenging vision task, dense video captioning, which aims to automatically describe a video clip with multiple informative and diverse caption sentences. The proposed method is trained without explicit annotation of fine-grained sentence to video region-sequence correspondence, but is only based on weak video-level sentence annotations. It differs from existing video captioning systems in three technical aspects. First, we propose lexical fully convolutional neural networks (Lexical-FCN) with weakly supervised multi-instance multi-label learning to weakly link video regions with lexical labels. Second, we introduce a novel submodular maximization scheme to generate multiple informative and diverse region-sequences based on the Lexical-FCN outputs. A winner-takes-all scheme is adopted to weakly associate sentences to region-sequences in the training phase. Third, a sequence-to-sequence learning based language model is trained with the weakly supervised information obtained through the association process. We show that the proposed method can not only produce informative and diverse dense captions, but also outperform state-of-the-art single video captioning methods by a large margin.
arXiv commentTo appear in CVPR 2017
References in corpus (22)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Sequence to Sequence Learning with Neural Networks
- Deep Residual Learning for Image Recognition
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Recurrent Neural Network Regularization
- Fully Convolutional Networks for Semantic Segmentation
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Coherent Multi-Sentence Video Description with Variable Level of Detail
- Sequence to Sequence -- Video to Text
- Describing Videos by Exploiting Temporal Structure
- Show and Tell: A Neural Image Caption Generator
- Learning Spatiotemporal Features with 3D Convolutional Networks
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Frame- and Segment-Level Features and Candidate Pool Evaluation for Video Caption Generation
- From Captions to Visual Concepts and Back
- Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
- CIDEr: Consensus-based Image Description Evaluation
- Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
- Jointly Modeling Embedding and Translation to Bridge Video and Language
- Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
- Beyond Caption To Narrative: Video Captioning With Multiple Sentences
Cited by in corpus (21)
- Reconstruction Network for Video Captioning
- Less Is More: Picking Informative Frames for Video Captioning
- Video Captioning via Hierarchical Reinforcement Learning
- Consensus-based Sequence Training for Video Captioning
- An Attempt towards Interpretable Audio-Visual Video Captioning
- Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
- Learning Visual Knowledge Memory Networks for Visual Question Answering
- Attend and Interact: Higher-Order Object Interactions for Video Understanding
- Video Captioning with Guidance of Multimodal Latent Topics
- Move Forward and Tell: A Progressive Generator of Video Descriptions
- LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
- End-to-End Video Captioning with Multitask Reinforcement Learning
- Grounded Objects and Interactions for Video Captioning
- Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
- Not All Words are Equal: Video-specific Information Loss for Video Captioning
- Multilevel Language and Vision Integration for Text-to-Clip Retrieval
- Mining for meaning: from vision to language through multiple networks consensus
- AutoLoc: Weakly-supervised Temporal Action Localization
- Multi-modal Dense Video Captioning
- DCA: Diversified Co-Attention towards Informative Live Video Commenting
- Live Video Comment Generation Based on Surrounding Frames and Live Comments