Reconstruction Network for Video Captioning

Abstract

In this paper, the problem of describing visual contents of a video sequence with natural language is addressed. Unlike previous video captioning work mainly exploiting the cues of video contents to make a language description, we propose a reconstruction network (RecNet) with a novel encoder-decoder-reconstructor architecture, which leverages both the forward (video to sentence) and backward (sentence to video) flows for video captioning. Specifically, the encoder-decoder makes use of the forward flow to produce the sentence description based on the encoded video semantic features. Two types of reconstructors are customized to employ the backward flow and reproduce the video features based on the hidden state sequence generated by the decoder. The generation loss yielded by the encoder-decoder and the reconstruction loss introduced by the reconstructor are jointly drawn into training the proposed RecNet in an end-to-end fashion. Experimental results on benchmark datasets demonstrate that the proposed reconstructor can boost the encoder-decoder models and leads to significant gains in video caption accuracy.

arXiv commentAccepted by CVPR 2018

References in corpus (18)

• Neural Machine Translation by Jointly Learning to Align and Translate
• ADADELTA: An Adaptive Learning Rate Method
• Microsoft COCO Captions: Data Collection and Evaluation Server
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Dual Learning for Machine Translation
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Delving Deeper into Convolutional Networks for Learning Video Representations
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• Neural Machine Translation with Reconstruction
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
• A Survey on Learning to Hash
• Weakly Supervised Dense Video Captioning
• Video Captioning with Transferred Semantic Attributes

Cited by in corpus (11)

• Multi-granularity Generator for Temporal Action Proposal
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
• An Attempt towards Interpretable Audio-Visual Video Captioning
• Non-local NetVLAD Encoding for Video Classification
• Streamlined Dense Video Captioning
• Spatio-temporal Video Re-localization by Warp LSTM
• Towards Diverse Paragraph Captioning for Untrimmed Videos
• Accurate Temporal Action Proposal Generation with Relation-Aware Pyramid Network
• Saying the Unseen: Video Descriptions via Dialog Agents
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• Deep hierarchical pooling design for cross-granularity action recognition