Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions

Abstract

We study joint video and language (VL) pre-training to enable cross-modality learning and benefit plentiful downstream VL tasks. Existing works either extract low-quality video features or learn limited text embedding, while neglecting that high-resolution videos and diversified semantics can significantly improve cross-modality learning. In this paper, we propose a novel High-resolution and Diversified VIdeo-LAnguage pre-training model (HD-VILA) for many visual tasks. In particular, we collect a large dataset with two distinct properties: 1) the first high-resolution dataset including 371.5k hours of 720p videos, and 2) the most diversified dataset covering 15 popular YouTube categories. To enable VL pre-training, we jointly optimize the HD-VILA model by a hybrid Transformer that learns rich spatiotemporal features, and a multimodal Transformer that enforces interactions of the learned video features with diversified texts. Our pre-training model achieves new state-of-the-art results in 10 VL understanding tasks and 2 more novel text-to-visual generation tasks. For example, we outperform SOTA models with relative increases of 40.4% R@1 in zero-shot MSR-VTT text-to-video retrieval task and 55.4% in high-resolution dataset LSMDC. The learned VL embedding is also effective in generating visually pleasing and semantically relevant results in text-to-visual editing and super-resolution tasks.

References in corpus (12)

• Decoupled Weight Decay Regularization
• Is Space-Time Attention All You Need for Video Understanding?
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Learning Video Representations using Contrastive Bidirectional Transformer
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Hadamard Product for Low-rank Bilinear Pooling
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• MERLOT: Multimodal Neural Script Knowledge Models
• S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networks
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

Cited by in corpus (1)

• STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training