Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data

Abstract

Scaling up weakly-supervised datasets has shown to be highly effective in the image-text domain and has contributed to most of the recent state-of-the-art computer vision and multimodal neural networks. However, existing large-scale video-text datasets and mining techniques suffer from several limitations, such as the scarcity of aligned data, the lack of diversity in the data, and the difficulty of collecting aligned data. Currently popular video-text data mining approach via automatic speech recognition (ASR) used in HowTo100M provides low-quality captions that often do not refer to the video content. Other mining approaches do not provide proper language descriptions (video tags) and are biased toward short clips (alt text). In this work, we show how recent advances in image captioning allow us to pre-train high-quality video models without any parallel video-text data. We pre-train several video captioning models that are based on an OPT language model and a TimeSformer visual backbone. We fine-tune these networks on several video captioning datasets. First, we demonstrate that image captioning pseudolabels work better for pre-training than the existing HowTo100M ASR captions. Second, we show that pre-training on both images and videos produces a significantly better network (+4 CIDER on MSR-VTT) than pre-training on a single modality. Our methods are complementary to the existing pre-training or data mining approaches and can be used in a variety of settings. Given the efficacy of the pseudolabeling method, we are planning to publicly release the generated captions.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PaLM: Scaling Language Modeling with Pathways
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Scaling Laws for Neural Language Models
• Is Space-Time Attention All You Need for Video Understanding?
• Flamingo: a Visual Language Model for Few-Shot Learning
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• The Curious Case of Neural Text Degeneration
• LAION-5B: An open large-scale dataset for training next generation image-text models
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• YouTube-8M: A Large-Scale Video Classification Benchmark
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Axial Attention in Multidimensional Transformers
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Attention Bottlenecks for Multimodal Fusion
• GIT: A Generative Image-to-text Transformer for Vision and Language
• Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
• SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
• Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training
• Distribution Aware Metrics for Conditional Natural Language Generation