VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
arXiv:2106.04632
Abstract
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.
To appear in 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks
References in corpus (7)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Learning Transferable Visual Models From Natural Language Supervision
- The Kinetics Human Action Video Dataset
- The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
- CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
- Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models