Video Fill in the Blank with Merging LSTMs
arXiv:1610.04062
Abstract
Given a video and its incomplete textural description with missing words, the Video-Fill-in-the-Blank (ViFitB) task is to automatically find the missing word. The contextual information of the sentences are important to infer the missing words; the visual cues are even more crucial to get a more accurate inference. In this paper, we presents a new method which intuitively takes advantage of the structure of the sentences and employs merging LSTMs (to merge two LSTMs) to tackle the problem with embedded textural and visual cues. In the experiments, we have demonstrated the superior performance of the proposed method on the challenging "Movie Fill-in-the-Blank" dataset.
arXiv commentfor Large Scale Movie Description and Understanding Challenge (LSMDC) 2016, "Movie fill-in-the-blank" Challenge, UCF_CRCV
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- VQA: Visual Question Answering
- Sequence to Sequence -- Video to Text
- Describing Videos by Exploiting Temporal Structure
- Visual Madlibs: Fill in the blank Image Generation and Question Answering
- Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
- Uncovering Temporal Context for Video Question and Answering
Cited by in corpus (5)
- End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
- A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering
- Open-Ended Long-Form Video Question Answering via Hierarchical Convolutional Self-Attention Networks
- ActBERT: Learning Global-Local Video-Text Representations
- The Forgettable-Watcher Model for Video Question Answering