Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning

Abstract

Dense video captioning is a newly emerging task that aims at both localizing and describing all events in a video. We identify and tackle two challenges on this task, namely, (1) how to utilize both past and future contexts for accurate event proposal predictions, and (2) how to construct informative input to the decoder for generating natural event descriptions. First, previous works predominantly generate temporal event proposals in the forward direction, which neglects future video context. We propose a bidirectional proposal method that effectively exploits both past and future contexts to make proposal predictions. Second, different events ending at (nearly) the same time are indistinguishable in the previous works, resulting in the same captions. We solve this problem by representing each event with an attentive fusion of hidden states from the proposal module and video contents (e.g., C3D features). We further propose a novel context gating mechanism to balance the contributions from the current event and its surrounding contexts dynamically. We empirically show that our attentively fused event representation is superior to the proposal hidden states or video contents alone. By coupling proposal and captioning modules into one unified framework, our model outperforms the state-of-the-arts on the ActivityNet Captions dataset with a relative gain of over 100% (Meteor score increases from 4.82 to 9.65).

arXiv commentCVPR2018 spotlight paper

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Show and Tell: A Neural Image Caption Generator
• Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
• CIDEr: Consensus-based Image Description Evaluation
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Boosting Image Captioning with Attributes
• Semantic Compositional Networks for Visual Captioning
• Video Captioning with Transferred Semantic Attributes
• Temporal Dynamic Graph LSTM for Action-driven Video Object Detection

Cited by in corpus (12)

• Multi-granularity Generator for Temporal Action Proposal
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
• Non-local NetVLAD Encoding for Video Classification
• A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer
• Streamlined Dense Video Captioning
• RUC+CMU: System Report for Dense Captioning Events in Videos
• Spatio-temporal Video Re-localization by Warp LSTM
• Semantic Adversarial Network with Multi-scale Pyramid Attention for Video Classification
• Towards Diverse Paragraph Captioning for Untrimmed Videos
• Multi-modal Dense Video Captioning
• Saying the Unseen: Video Descriptions via Dialog Agents
• Deep hierarchical pooling design for cross-granularity action recognition