Complete the Look: Scene-based Complementary Product Recommendation
arXiv:1812.01748
Abstract
Modeling fashion compatibility is challenging due to its complexity and subjectivity. Existing work focuses on predicting compatibility between product images (e.g. an image containing a t-shirt and an image containing a pair of jeans). However, these approaches ignore real-world 'scene' images (e.g. selfies); such images are hard to deal with due to their complexity, clutter, variations in lighting and pose (etc.) but on the other hand could potentially provide key context (e.g. the user's body type, or the season) for making more accurate recommendations. In this work, we propose a new task called 'Complete the Look', which seeks to recommend visually compatible products based on scene images. We design an approach to extract training data for this task, and propose a novel way to learn the scene-product compatibility from fashion or interior design images. Our approach measures compatibility both globally and locally via CNNs and attention mechanisms. Extensive experiments show that our method achieves significant performance gains over alternative systems. Human evaluation and qualitative analysis are also conducted to further understand model behavior. We hope this work could lead to useful applications which link large corpora of real-world scenes with shoppable products.
Accepted to CVPR'19
References in corpus (13)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- BPR: Bayesian Personalized Ranking from Implicit Feedback
- Self-Attention Generative Adversarial Networks
- CNN Features off-the-shelf: an Astounding Baseline for Recognition
- DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection
- Visual Search at eBay
- Inferring Networks of Substitutable and Complementary Products
- Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences
- Automatic Spatially-aware Fashion Concept Discovery
- Compatibility Family Learning for Item Recommendation and Generation
- ModaNet: A Large-Scale Street Fashion Dataset with Polygon Annotations
- Learning Type-Aware Embeddings for Fashion Compatibility
- Fashion Forward: Forecasting Visual Style in Fashion