GuessWhat?! Visual object discovery through multi-modal dialogue

Abstract

We introduce GuessWhat?!, a two-player guessing game as a testbed for research on the interplay of computer vision and dialogue systems. The goal of the game is to locate an unknown object in a rich image scene by asking a sequence of questions. Higher-level image understanding, like spatial reasoning and language grounding, is required to solve the proposed task. Our key contribution is the collection of a large-scale dataset consisting of 150K human-played games with a total of 800K visual question-answer pairs on 66K images. We explain our design decisions in collecting the dataset and introduce the oracle and questioner tasks that are associated with the two players of the game. We prototyped deep learning models to establish initial baselines of the introduced tasks.

23 pages; CVPR 2017 submission; see https://guesswhat.ai

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Stacked Attention Networks for Image Question Answering
• Show and Tell: A Neural Image Caption Generator
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems
• Modeling Context in Referring Expressions

Cited by in corpus (9)

• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
• Discriminability objective for training descriptive captions
• C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset
• Emergent Communication in a Multi-Modal, Multi-Step Referential Game
• Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewards
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• A System for Automated Image Editing from Natural Language Commands
• Learning to Disambiguate by Asking Discriminative Questions