Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training

Abstract

In this work, we present Auto-captions on GIF, which is a new large-scale pre-training dataset for generic video understanding. All video-sentence pairs are created by automatically extracting and filtering video caption annotations from billions of web pages. Auto-captions on GIF dataset can be utilized to pre-train the generic feature representation or encoder-decoder structure for video captioning, and other downstream tasks (e.g., sentence localization in videos, video question answering, etc.) as well. We present a detailed analysis of Auto-captions on GIF dataset in comparison to existing video-sentence datasets. We also provide an evaluation of a Transformer-based encoder-decoder structure for vision-language pre-training, which is further adapted to video captioning downstream task and yields the compelling generalizability on MSR-VTT. The dataset is available at \url{http://www.auto-video-captions.top/2020/dataset}.

arXiv commentThe Auto-captions on GIF dataset is available at \url{http://www.auto-video-captions.top/2020/dataset}

References in corpus (7)

• NLTK: The Natural Language Toolkit
• Coherent Multi-Sentence Video Description with Variable Level of Detail
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• TGIF: A New Dataset and Benchmark on Animated GIF Description
• Jointly Localizing and Describing Events for Dense Video Captioning
• Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
• Convolutional Auto-encoding of Sentence Topics for Image Paragraph Generation

Cited by in corpus (5)

• Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval
• Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• Survey: Transformer based Video-Language Pre-training
• CLIP4Caption: CLIP for Video Caption