Video Captioning with Transferred Semantic Attributes

Abstract

Automatically generating natural language descriptions of videos plays a fundamental challenge for computer vision community. Most recent progress in this problem has been achieved through employing 2-D and/or 3-D Convolutional Neural Networks (CNN) to encode video content and Recurrent Neural Networks (RNN) to decode a sentence. In this paper, we present Long Short-Term Memory with Transferred Semantic Attributes (LSTM-TSA)---a novel deep architecture that incorporates the transferred semantic attributes learnt from images and videos into the CNN plus RNN framework, by training them in an end-to-end manner. The design of LSTM-TSA is highly inspired by the facts that 1) semantic attributes play a significant contribution to captioning, and 2) images and videos carry complementary semantics and thus can reinforce each other for captioning. To boost video captioning, we propose a novel transfer unit to model the mutually correlated attributes learnt from images and videos. Extensive experiments are conducted on three public datasets, i.e., MSVD, M-VAD and MPII-MD. Our proposed LSTM-TSA achieves to-date the best published performance in sentence generation on MSVD: 52.8% and 74.0% in terms of BLEU@4 and CIDEr-D. Superior results when compared to state-of-the-art methods are also reported on M-VAD and MPII-MD.

References in corpus (9)

• Microsoft COCO Captions: Data Collection and Evaluation Server
• Explain Images with Multimodal Recurrent Neural Networks
• Image Captioning with Semantic Attention
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• A Dataset for Movie Description
• Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
• Rich Image Captioning in the Wild

Cited by in corpus (11)

• End-to-End Dense Video Captioning with Masked Transformer
• Reconstruction Network for Video Captioning
• Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning
• Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
• Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
• Hierarchical LSTMs with Adaptive Attention for Visual Captioning
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Move Forward and Tell: A Progressive Generator of Video Descriptions
• Grounded Objects and Interactions for Video Captioning
• MTLE: A Multitask Learning Encoder of Visual Feature Representations for Video and Movie Description
• Grounded Video Description