VIOLIN: A Large-Scale Dataset for Video-and-Language Inference

Abstract

We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text. Given a video clip with aligned subtitles as premise, paired with a natural language hypothesis based on the video content, a model needs to infer whether the hypothesis is entailed or contradicted by the given video clip. A new large-scale dataset, named Violin (VIdeO-and-Language INference), is introduced for this task, which consists of 95,322 video-hypothesis pairs from 15,887 video clips, spanning over 582 hours of video. These video clips contain rich content with diverse temporal dynamics, event shifts, and people interactions, collected from two sources: (i) popular TV shows, and (ii) movie clips from YouTube channels. In order to address our new multimodal inference task, a model is required to possess sophisticated reasoning skills, from surface-level grounding (e.g., identifying objects and characters in the video) to in-depth commonsense reasoning (e.g., inferring causal relations of events in the video). We present a detailed analysis of the dataset and an extensive evaluation over many strong baselines, providing valuable insights on the challenges of this new task.

arXiv commentAccepted to CVPR2020

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• The Kinetics Human Action Video Dataset
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Bidirectional Attention Flow for Machine Comprehension
• YouTube-8M: A Large-Scale Video Classification Benchmark
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Multi-Task Deep Neural Networks for Natural Language Understanding
• UNITER: UNiversal Image-TExt Representation Learning
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Compositional Attention Networks for Machine Reasoning
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval
• Meta Module Network for Compositional Visual Reasoning

Cited by in corpus (5)

• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Fine-grained Iterative Attention Network for TemporalLanguage Localization in Videos
• Winning the ICCV'2021 VALUE Challenge: Task-aware Ensemble and Transfer Learning with Visual Concepts
• Situation and Behavior Understanding by Trope Detection on Films
• TrUMAn: Trope Understanding in Movies and Animations