VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation

Abstract

Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.

arXiv commentTo appear in 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks

References in corpus (12)

• Decoupled Weight Decay Regularization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Learning Transferable Visual Models From Natural Language Supervision
• The Kinetics Human Action Video Dataset
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Towards Automatic Learning of Procedures from Web Instructional Videos
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
• CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding

Cited by in corpus (12)

• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
• VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
• A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• Relevance-based Margin for Contrastively-trained Video Retrieval Models
• SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
• Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
• Survey: Transformer based Video-Language Pre-training
• Coarse to Fine: Video Retrieval before Moment Localization
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations