Survey: Transformer based Video-Language Pre-training
arXiv:2109.09920
Abstract
Inspired by the success of transformer-based pre-training methods on natural language tasks and further computer vision tasks, researchers have begun to apply transformer to video processing. This survey aims to give a comprehensive overview on transformer-based pre-training methods for Video-Language learning. We first briefly introduce the transformer tructure as the background knowledge, including attention mechanism, position encoding etc. We then describe the typical paradigm of pre-training & fine-tuning on Video-Language processing in terms of proxy tasks, downstream tasks and commonly used video datasets. Next, we categorize transformer models into Single-Stream and Multi-Stream structures, highlight their innovations and compare their performances. Finally, we analyze and discuss the current challenges and possible future research directions for Video-Language pre-training.
References in corpus (9)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- The Kinetics Human Action Video Dataset
- Microsoft COCO Captions: Data Collection and Evaluation Server
- A Comprehensive Study of Deep Video Action Recognition
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
- VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
- Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training
- CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning