Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning

Abstract

Recent progress has been made in using attention based encoder-decoder framework for video captioning. However, most existing decoders apply the attention mechanism to every generated word including both visual words (e.g., "gun" and "shooting") and non-visual words (e.g. "the", "a"). However, these non-visual words can be easily predicted using natural language model without considering visual signals or attention. Imposing attention mechanism on non-visual words could mislead and decrease the overall performance of video captioning. To address this issue, we propose a hierarchical LSTM with adjusted temporal attention (hLSTMat) approach for video captioning. Specifically, the proposed framework utilizes the temporal attention for selecting specific frames to predict the related words, while the adjusted temporal attention is for deciding whether to depend on the visual information or the language context information. Also, a hierarchical LSTMs is designed to simultaneously consider both low-level visual information and high-level language context information to support the video caption generation. To demonstrate the effectiveness of our proposed framework, we test our method on two prevalent datasets: MSVD and MSR-VTT, and experimental results show that our approach outperforms the state-of-the-art methods on both two datasets.

References in corpus (8)

• ADADELTA: An Adaptive Learning Rate Method
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Sequence to Sequence -- Video to Text
• Learning a Recurrent Visual Representation for Image Caption Generation
• Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Jointly Modeling Embedding and Translation to Bridge Video and Language
• Semantic Compositional Networks for Visual Captioning

Cited by in corpus (14)

• Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Reconstruction Network for Video Captioning
• Video Captioning via Hierarchical Reinforcement Learning
• Memory-Attended Recurrent Network for Video Captioning
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
• Grounded Objects and Interactions for Video Captioning
• In-Home Daily-Life Captioning Using Radio Signals
• Not All Words are Equal: Video-specific Information Loss for Video Captioning
• Video Captioning with Text-based Dynamic Attention and Step-by-Step Learning
• Semantic Grouping Network for Video Captioning
• Guided Attention for Interpretable Motion Captioning
• Video Captioning in Compressed Video