A Dataset for Movie Description
arXiv:1501.02530
Abstract
Descriptive video service (DVS) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed DVS, which is temporally aligned to full length HD movies. In addition we also collected the aligned movie scripts which have been used in prior work and compare the two different sources of descriptions. In total the Movie Description dataset contains a parallel corpus of over 54,000 sentences and video snippets from 72 HD movies. We characterize the dataset by benchmarking different approaches for generating video descriptions. Comparing DVS to scripts, we find that DVS is far more visual and describes precisely what is shown rather than what should happen according to the scripts created prior to movie production.
References in corpus (9)
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Coherent Multi-Sentence Video Description with Variable Level of Detail
- Show and Tell: A Neural Image Caption Generator
- Learning a Recurrent Visual Representation for Image Caption Generation
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- From Captions to Visual Concepts and Back
- Weakly Supervised Action Labeling in Videos Under Ordering Constraints
- Video In Sentences Out
Cited by in corpus (29)
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
- Sequence to Sequence -- Video to Text
- MDMMT: Multidomain Multimodal Transformer for Video Retrieval
- Frame- and Segment-Level Features and Candidate Pool Evaluation for Video Caption Generation
- TVQA: Localized, Compositional Video Question Answering
- DeepStory: Video Story QA by Deep Embedded Memory Networks
- Video captioning with recurrent networks based on frame- and video-level features and visual content classification
- Stories for Images-in-Sequence by using Visual and Narrative Components
- Generating Descriptions with Grounded and Co-Referenced People
- Video Captioning with Transferred Semantic Attributes
- Multimodal Visual Concept Learning with Weakly Supervised Techniques
- Integrating both Visual and Audio Cues for Enhanced Video Caption
- Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
- Title Generation for User Generated Videos
- Video Highlight Prediction Using Audience Chat Reactions
- MarioQA: Answering Questions by Watching Gameplay Videos
- LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
- A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering
- Temporal Tessellation: A Unified Approach for Video Analysis
- MTLE: A Multitask Learning Encoder of Visual Feature Representations for Video and Movie Description
- Embodied Language Grounding with 3D Visual Feature Representations
- ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
- Learning Action Concept Trees and Semantic Alignment Networks from Image-Description Data
- VideoMCC: a New Benchmark for Video Comprehension
- ODSQA: Open-domain Spoken Question Answering Dataset
- NarrationBot and InfoBot: A Hybrid System for Automated Video Description
- The Long-Short Story of Movie Description