TGIF: A New Dataset and Benchmark on Animated GIF Description

Abstract

With the recent popularity of animated GIFs on social media, there is need for ways to index them with rich metadata. To advance research on animated GIF understanding, we collected a new dataset, Tumblr GIF (TGIF), with 100K animated GIFs from Tumblr and 120K natural language descriptions obtained via crowdsourcing. The motivation for this work is to develop a testbed for image sequence description systems, where the task is to generate natural language descriptions for animated GIFs or video clips. To ensure a high quality dataset, we developed a series of novel quality controls to validate free-form text input from crowdworkers. We show that there is unambiguous association between visual content and natural language descriptions in our dataset, making it an ideal benchmark for the visual content captioning task. We perform extensive statistical analyses to compare our dataset to existing image and video description datasets. Next, we provide baseline results on the animated GIF description task, using three representative techniques: nearest neighbor, statistical machine translation, and recurrent neural networks. Finally, we show that models fine-tuned from our animated GIF description dataset can be helpful for automatic movie description.

arXiv commentCVPR 2016 Camera Ready

References in corpus (4)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• VQA: Visual Question Answering
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• Jointly Modeling Embedding and Translation to Bridge Video and Language

Cited by in corpus (11)

• TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
• Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
• DeepStory: Video Story QA by Deep Embedded Memory Networks
• Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training
• Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
• Cross-Modal Retrieval with Implicit Concept Association
• Dual Encoding for Zero-Example Video Retrieval
• Streamlined Dense Video Captioning
• Video Captioning with Guidance of Multimodal Latent Topics
• A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks
• The Forgettable-Watcher Model for Video Question Answering