Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
arXiv:1905.12794
Abstract
Conversational interfaces for the detail-oriented retail fashion domain are more natural, expressive, and user friendly than classical keyword-based search interfaces. In this paper, we introduce the Fashion IQ dataset to support and advance research on interactive fashion image retrieval. Fashion IQ is the first fashion dataset to provide human-generated captions that distinguish similar pairs of garment images together with side-information consisting of real-world product descriptions and derived visual attribute labels for these images. We provide a detailed analysis of the characteristics of the Fashion IQ data, and present a transformer-based user simulator and interactive image retriever that can seamlessly integrate visual attributes with image features, user feedback, and dialog history, leading to improved performance over the state of the art in dialog-based image retrieval. We believe that our dataset will encourage further work on developing more natural and real-world applicable conversational shopping assistants.
References in corpus (13)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- VQA: Visual Question Answering
- Towards Deep Conversational Recommendations
- Visual Search at eBay
- Dialog-based Interactive Image Retrieval
- Fashion-Gen: The Generative Fashion Dataset and Challenge
- Modality-Agnostic Attention Fusion for visual search with text feedback
- ModaNet: A Large-Scale Street Fashion Dataset with Polygon Annotations
- DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images
- Drill-down: Interactive Retrieval of Complex Scenes using Natural Language Queries
- Expressing Visual Relationships via Language
Cited by in corpus (9)
- Modality-Agnostic Attention Fusion for visual search with text feedback
- RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network
- Sentiment Analysis of Fashion Related Posts in Social Media
- Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards
- Reward Constrained Interactive Recommendation with Natural Language Feedback
- Designovel's system description for Fashion-IQ challenge 2019
- Arbitrary Virtual Try-On Network: Characteristics Preservation and Trade-off between Body and Clothing
- Conversational Fashion Image Retrieval via Multiturn Natural Language Feedback
- Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training