Video Understanding as Machine Translation

Abstract

With the advent of large-scale multimodal video datasets, especially sequences with audio or transcribed speech, there has been a growing interest in self-supervised learning of video representations. Most prior work formulates the objective as a contrastive metric learning problem between the modalities. To enable effective learning, however, these strategies require a careful selection of positive and negative samples often combined with hand-designed curriculum policies. In this work we remove the need for negative sampling by taking a generative modeling approach that poses the objective as a translation problem between modalities. Such a formulation allows us to tackle a wide variety of downstream video understanding tasks by means of a single unified framework, without the need for large batches of negative samples common in contrastive metric learning. We experiment with the large-scale HowTo100M dataset for training, and report performance gains over the state-of-the-art on several downstream tasks including video classification (EPIC-Kitchens), question answering (TVQA), captioning (TVC, YouCook2, and MSR-VTT), and text-based clip retrieval (YouCook2 and MSR-VTT).

The authors have temporarily withdrawn this paper to reassess some of the experimental results

References in corpus (17)

• A Simple Framework for Contrastive Learning of Visual Representations
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Contrastive Multiview Coding
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• How Context Affects Language Models' Factual Predictions
• Video Representation Learning by Dense Predictive Coding
• Look, Listen and Learn
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
• Document Ranking with a Pretrained Sequence-to-Sequence Model

Cited by in corpus (6)

• MusCaps: Generating Captions for Music Audio
• Support-Set Based Cross-Supervision for Video Grounding
• Video-Text Pre-training with Learned Regions
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
• VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models