UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
arXiv:2002.06353
Abstract
With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL: a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder, and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks.
Cited by in corpus (10)
- VLP: A Survey on Vision-Language Pre-training
- Self-Supervised Learning for Videos: A Survey
- Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval
- A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes
- RTQ: Rethinking Video-language Understanding Based on Image-text Model
- Multi-granularity Contrastive Cross-modal Collaborative Generation for End-to-End Long-term Video Question Answering
- STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
- Saturn Platform: Foundation Model Operations and Generative AI for Financial Services
- SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
- CLOP: Video-and-Language Pre-Training with Knowledge Regularizations