VideoBERT: A Joint Model for Video and Language Representation Learning

Abstract

Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.

arXiv commentICCV 2019 camera ready

References in corpus (11)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• The Kinetics Human Action Video Dataset
• Cross-lingual Language Model Pretraining
• Going Deeper with Convolutions
• Stochastic Video Generation with a Learned Prior
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Stochastic Adversarial Video Prediction
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interaction
• End-to-End Dense Video Captioning with Masked Transformer

Cited by in corpus (56)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Attention Bottlenecks for Multimodal Fusion
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Unifying Vision-and-Language Tasks via Text Generation
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Self-Supervised Learning for Videos: A Survey
• MDMMT: Multidomain Multimodal Transformer for Video Retrieval
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Materials science in the era of large language models: a perspective
• MERLOT: Multimodal Neural Script Knowledge Models
• MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
• Dynamic Neural Networks: A Survey
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Discrete and continuous representations and processing in deep learning: Looking forward
• Video Understanding as Machine Translation
• Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog
• Global Self-Attention Networks for Image Recognition
• Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
• Pre-Trained Models: Past, Present and Future
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
• Variable-rate discrete representation learning
• Integrating Multimodal Information in Large Pretrained Transformers
• Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
• Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
• Towards Debiasing Sentence Representations
• STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
• Learning Potentials of Quantum Systems using Deep Neural Networks
• Temporal Query Networks for Fine-grained Video Understanding
• Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
• A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• Cross-modal Learning for Multi-modal Video Categorization
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
• Context-Aware Group Captioning via Self-Attention and Contrastive Features
• CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising
• SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
• A Visuospatial Dataset for Naturalistic Verb Learning
• Human-like general language processing
• Probing Contextualized Sentence Representations with Visual Awareness
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• Answer-checking in Context: A Multi-modal FullyAttention Network for Visual Question Answering
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• On Compositions of Transformations in Contrastive Self-Supervised Learning
• Accurate Word Representations with Universal Visual Guidance
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions