End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering

Abstract

We propose a high-level concept word detector that can be integrated with any video-to-language models. It takes a video as input and generates a list of concept words as useful semantic priors for language generation models. The proposed word detector has two important properties. First, it does not require any external knowledge sources for training. Second, the proposed word detector is trainable in an end-to-end manner jointly with any video-to-language models. To maximize the values of detected words, we also develop a semantic attention mechanism that selectively focuses on the detected concept words and fuse them with the word encoding and decoding in the language model. In order to demonstrate that the proposed approach indeed improves the performance of multiple video-to-language tasks, we participate in four tasks of LSMDC 2016. Our approach achieves the best accuracies in three of them, including fill-in-the-blank, multiple-choice test, and movie retrieval. We also attain comparable performance for the other task, movie description.

In CVPR 2017. Winner of three (fill-in-the-blank, multiple-choice test, and movie retrieval) out of four tasks of the LSMDC 2016 Challenge. 22 pages

References in corpus (5)

• Layer Normalization
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language
• Video Fill in the Blank with Merging LSTMs
• Temporal Tessellation: A Unified Approach for Video Analysis

Cited by in corpus (14)

• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Video Captioning via Hierarchical Reinforcement Learning
• A Read-Write Memory Network for Movie Story Understanding
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering
• Encoding Video and Label Priors for Multi-label Video Classification on YouTube-8M dataset
• Temporal Tessellation: A Unified Approach for Video Analysis
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• SSAN: Separable Self-Attention Network for Video Representation Learning
• Transfer Learning in Visual and Relational Reasoning
• Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA
• Towards Diverse Paragraph Captioning for Untrimmed Videos