VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling

Abstract

A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. Recent studies try to mitigate this disconnection via end-to-end training. To make it computationally feasible, prior works tend to "imagify" video inputs, i.e., a handful of sparsely sampled frames are fed into a 2D CNN, followed by a simple mean-pooling or concatenation to obtain the overall video representations. Although achieving promising results, such simple approaches may lose temporal information that is essential for performing downstream VidL tasks. In this work, we present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs. Further, unlike previous studies that found pre-training tasks on video inputs (e.g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling. Specifically, the original video frame patches are "tokenized" into discrete visual tokens, and the goal is to recover the original visual tokens based on the masked patches. Comprehensive analysis demonstrates the effectiveness of both explicit temporal modeling via video transformer and MVM. As a result, VIOLET achieves new state-of-the-art performance on 5 video question answering tasks and 4 text-to-video retrieval tasks.

Code is available at https://github.com/tsujuifu/pytorch_violet

References in corpus (14)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Transferable Visual Models From Natural Language Supervision
• The Kinetics Human Action Video Dataset
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Is Space-Time Attention All You Need for Video Understanding?
• Zero-Shot Text-to-Image Generation
• BEiT: BERT Pre-Training of Image Transformers
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language
• Video Swin Transformer
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
• QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries

Cited by in corpus (4)

• Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
• STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
• SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks