VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
arXiv:2109.14084
Abstract
We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.
EMNLP 2021
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Learning Transferable Visual Models From Natural Language Supervision
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Zero-Shot Text-to-Image Generation
- Learning Language-Visual Embedding for Movie Understanding with Natural-Language
- Video Understanding as Machine Translation
- Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models