Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
arXiv:2302.14115
Abstract
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale. The Vid2Seq architecture augments a language model with special time tokens, allowing it to seamlessly predict event boundaries and textual descriptions in the same output sequence. Such a unified model requires large-scale training data, which is not available in current annotated datasets. We show that it is possible to leverage unlabeled narrated videos for dense video captioning, by reformulating sentence boundaries of transcribed speech as pseudo event boundaries, and using the transcribed speech sentences as pseudo event captions. The resulting Vid2Seq model pretrained on the YT-Temporal-1B dataset improves the state of the art on a variety of dense video captioning benchmarks including YouCook2, ViTT and ActivityNet Captions. Vid2Seq also generalizes well to the tasks of video paragraph captioning and video clip captioning, and to few-shot settings. Our code is publicly available at https://antoyang.github.io/vid2seq.html.
arXiv commentCVPR 2023 Camera-Ready; Project Webpage: https://antoyang.github.io/vid2seq.html ; 18 pages; 6 figures
References in corpus (39)
- Adam: A Method for Stochastic Optimization
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Flamingo: a Visual Language Model for Few-Shot Learning
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- YouTube-8M: A Large-Scale Video Classification Benchmark
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- CoCa: Contrastive Captioners are Image-Text Foundation Models
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- Florence: A New Foundation Model for Computer Vision
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
- End-to-end Temporal Action Detection with Transformer
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- PaLI: A Jointly-Scaled Multilingual Language-Image Model
- WikiHow: A Large Scale Text Summarization Dataset
- SeqTR: A Simple yet Universal Network for Visual Grounding
- Pix2seq: A Language Modeling Framework for Object Detection
- GLIPv2: Unifying Localization and Vision-Language Understanding
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
- VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
- Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
- Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
- A Unified Sequence Interface for Vision Tasks
- Egocentric Video-Language Pretraining
- VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
- Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning
- SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
- UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
- iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering
- UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes
- End-to-end Dense Video Captioning as Sequence Generation
- Multimodal Pretraining for Dense Video Captioning
- Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks
- Contrastive Video-Language Learning with Fine-grained Frame Sampling
- Contrastive Language-Action Pre-training for Temporal Localization