Visual Madlibs: Fill in the blank Image Generation and Question Answering

Abstract

In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-blank templates designed to gather targeted descriptions about: people and objects, their appearances, activities, and interactions, as well as inferences about the general scene or its broader context. We provide several analyses of the Visual Madlibs dataset and demonstrate its applicability to two new description generation tasks: focused description generation, and multiple-choice question-answering for images. Experiments using joint-embedding and deep learning methods show promising results on these tasks.

arXiv comment10 pages; 8 figures; 4 tables

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Natural Language Processing (almost) from Scratch
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Learning a Recurrent Visual Representation for Image Caption Generation
• Question Answering with Subgraph Embeddings
• From Captions to Visual Concepts and Back
• Open Question Answering with Weakly Supervised Embedding Models
• Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks

Cited by in corpus (21)

• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Learning Deep Structure-Preserving Image-Text Embeddings
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• TVQA: Localized, Compositional Video Question Answering
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Visual Question Answering: A Survey of Methods and Datasets
• Visual7W: Grounded Question Answering in Images
• Uncovering Temporal Context for Video Question and Answering
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Revisiting Visual Question Answering Baselines
• Video Fill in the Blank with Merging LSTMs
• VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• Mean Box Pooling: A Rich Image Representation and Output Embedding for the Visual Madlibs Task
• Visual Question: Predicting If a Crowd Will Agree on the Answer
• Disjoint Multi-task Learning between Heterogeneous Human-centric Tasks
• Visual Question Answering as a Multi-Task Problem
• Leveraging Visual Question Answering for Image-Caption Ranking
• Reasoning Over History: Context Aware Visual Dialog