Modality-Agnostic Attention Fusion for visual search with text feedback

Abstract

Image retrieval with natural language feedback offers the promise of catalog search based on fine-grained visual features that go beyond objects and binary attributes, facilitating real-world applications such as e-commerce. Our Modality-Agnostic Attention Fusion (MAAF) model combines image and text features and outperforms existing approaches on two visual search with modifying phrase datasets, Fashion IQ and CSS, and performs competitively on a dataset with only single-word modifications, Fashion200k. We also introduce two new challenging benchmarks adapted from Birds-to-Words and Spot-the-Diff, which provide new settings with rich language inputs, and we show that our approach without modification outperforms strong baselines. To better understand our model, we conduct detailed ablations on Fashion IQ and provide visualizations of the surprising phenomenon of words avoiding "attending" to the image region they refer to.

arXiv comment14 pages, 8 figures

References in corpus (14)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• In Defense of the Triplet Loss for Person Re-Identification
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• CosFace: Large Margin Cosine Loss for Deep Face Recognition
• UNITER: UNiversal Image-TExt Representation Learning
• Review Networks for Caption Generation
• Unified Vision-Language Pre-Training for Image Captioning and VQA
• Automatic Spatially-aware Fashion Concept Discovery
• Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations
• Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
• CurlingNet: Compositional Learning between Images and Text for Fashion IQ Data
• FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval
• Designovel's system description for Fashion-IQ challenge 2019

Cited by in corpus (5)

• Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
• RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network
• CoVR-2: Automatic Data Construction for Composed Video Retrieval
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models