VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research

Abstract

We present a new large-scale multilingual video description dataset, VATEX, which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset, VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VATEX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VATEX for other video-and-language research.

arXiv commentICCV 2019 Oral. 17 pages, 14 figures, 6 tables (updated the VATEX website link: vatex-challenge.org)

References in corpus (15)

• Sequence to Sequence Learning with Neural Networks
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• VQA: Visual Question Answering
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• Towards Automatic Learning of Procedures from Web Instructional Videos
• How2: A Large-scale Dataset for Multimodal Language Understanding
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• Fluency-Guided Cross-Lingual Image Captioning
• Multimodal Attention for Neural Machine Translation
• Incorporating Global Visual Features into Attention-Based Neural Machine Translation
• Using Artificial Tokens to Control Languages for Multilingual Image Caption Generation

Cited by in corpus (10)

• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Normalized and Geometry-Aware Self-Attention Network for Image Captioning
• On Grounded Planning for Embodied Tasks with Language Models
• Coimagining the Future of Voice Assistants with Cultural Sensitivity
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• Rudder: A Cross Lingual Video and Text Retrieval Dataset
• Towards Diverse Paragraph Captioning for Untrimmed Videos
• Integrating Temporal and Spatial Attentions for VATEX Video Captioning Challenge 2019
• VATEX Captioning Challenge 2019: Multi-modal Information Fusion and Multi-stage Training Strategy for Video Captioning
• Imperial College London Submission to VATEX Video Captioning Task