Dialog-based Interactive Image Retrieval

Abstract

Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement learning problem, and reward the dialog system for improving the rank of the target image during each dialog turn. To mitigate the cumbersome and costly process of collecting human-machine conversations as the dialog system learns, we train our system with a user simulator, which is itself trained to describe the differences between target and candidate images. The efficacy of our approach is demonstrated in a footwear retrieval application. Experiments on both simulated and real-world data show that 1) our proposed learning framework achieves better accuracy than other supervised and reinforcement learning baselines and 2) user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results, and a more natural and expressive communication interface.

accepted at NeurIPS 2018

References in corpus (7)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• A User Simulator for Task-Completion Dialogues
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Person Search with Natural Language Description
• End-to-end optimization of goal-driven and visually grounded dialogue systems
• Context-aware Captions from Context-agnostic Supervision

Cited by in corpus (13)

• Geo-Localization via Ground-to-Satellite Cross-View Image Retrieval
• Fine-Grained Fashion Similarity Prediction by Attribute-Specific Embedding Learning
• Text as Neural Operator: Image Manipulation by Text Instruction
• Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
• RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network
• Fashion Retrieval via Graph Reasoning Networks on a Similarity Pyramid
• Interactive Interior Design Recommendation via Coarse-to-fine Multimodal Reinforcement Learning
• Sequential Attention GAN for Interactive Image Editing
• Interactive Video Corpus Moment Retrieval using Reinforcement Learning
• Interactive Classification by Asking Informative Questions
• ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval
• Composing Text and Image for Image Retrieval - An Empirical Odyssey
• Let's Transfer Transformations of Shared Semantic Representations