MERLOT: Multimodal Neural Script Knowledge Models

Abstract

As humans, we understand events in the visual world contextually, performing multimodal reasoning across time to make inferences about the past, present, and future. We introduce MERLOT, a model that learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech -- in an entirely label-free, self-supervised manner. By pretraining with a mix of both frame-level (spatial) and video-level (temporal) objectives, our model not only learns to match images to temporally corresponding words, but also to contextualize what is happening globally over time. As a result, MERLOT exhibits strong out-of-the-box representations of temporal commonsense, and achieves state-of-the-art performance on 12 different video QA datasets when finetuned. It also transfers well to the world of static images, allowing models to reason about the dynamic context behind visual scenes. On Visual Commonsense Reasoning, MERLOT answers questions correctly with 80.6% accuracy, outperforming state-of-the-art models of similar size by over 3%, even those that make heavy use of auxiliary supervised data (like object bounding boxes). Ablation analyses demonstrate the complementary importance of: 1) training on videos versus static images; 2) scaling the magnitude and diversity of the pretraining video corpus; and 3) using diverse objectives that encourage full-stack multimodal reasoning, from the recognition to cognition level.

arXiv commentproject page at https://rowanzellers.com/merlot; NeurIPS 2021 camera ready

References in corpus (29)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning Transferable Visual Models From Natural Language Supervision
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• On the Opportunities and Risks of Foundation Models
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• The Curious Case of Neural Text Degeneration
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Interaction Networks for Learning about Objects, Relations and Physics
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• Learning Video Representations using Contrastive Bidirectional Transformer
• Self-Supervised MultiModal Versatile Networks
• UNITER: UNiversal Image-TExt Representation Learning
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• VideoBERT: A Joint Model for Video and Language Representation Learning
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning
• Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data
• DramaQA: Character-Centered Video Story Understanding with Hierarchical QA
• Disembodied Machine Learning: On the Illusion of Objectivity in NLP
• Grounding Object Detections With Transcriptions
• Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
• What is More Likely to Happen Next? Video-and-Language Future Event Prediction
• Learning Temporal Dynamics from Cycles in Narrated Video

Cited by in corpus (7)

• On the Opportunities and Risks of Foundation Models
• Evaluating Large Language Models Trained on Code
• Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval
• PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting
• STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training
• Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
• Data Efficient Masked Language Modeling for Vision and Language