Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
arXiv:1503.01070
Abstract
In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DVS-derived dataset with minimal human intervention. Using this method, we have collected the largest DVS-derived dataset for video description of which we are aware. Our dataset currently includes over 84.6 hours of paired video/sentences from 92 DVDs and is growing.
7 pages
References in corpus (2)
Cited by in corpus (34)
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
- Sequence to Sequence -- Video to Text
- Describing Videos by Exploiting Temporal Structure
- Multimodal Machine Learning: A Survey and Taxonomy
- MDMMT: Multidomain Multimodal Transformer for Video Retrieval
- Frame- and Segment-Level Features and Candidate Pool Evaluation for Video Caption Generation
- TGIF: A New Dataset and Benchmark on Animated GIF Description
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
- VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
- DeepStory: Video Story QA by Deep Embedded Memory Networks
- Video Action Understanding
- M-VAD Names: a Dataset for Video Captioning with Naming
- Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training
- Stories for Images-in-Sequence by using Visual and Narrative Components
- Recent Advances and Trends in Multimodal Deep Learning: A Review
- Generating Descriptions with Grounded and Co-Referenced People
- Hierarchical LSTMs with Adaptive Attention for Visual Captioning
- Video Captioning with Transferred Semantic Attributes
- Multimodal Visual Concept Learning with Weakly Supervised Techniques
- Title Generation for User Generated Videos
- Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies
- Video Highlight Prediction Using Audience Chat Reactions
- CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
- LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
- Beyond Caption To Narrative: Video Captioning With Multiple Sentences
- MTLE: A Multitask Learning Encoder of Visual Feature Representations for Video and Movie Description
- MSVD-Turkish: A Comprehensive Multimodal Dataset for Integrated Vision and Language Research in Turkish
- A Comprehensive Review on Recent Methods and Challenges of Video Description
- Learning Action Concept Trees and Semantic Alignment Networks from Image-Description Data
- VideoMCC: a New Benchmark for Video Comprehension
- The Role of the Input in Natural Language Video Description
- The Long-Short Story of Movie Description
- Narration Generation for Cartoon Videos