Mining Fashion Outfit Composition Using An End-to-End Deep Learning Approach on Set Data

Abstract

Composing fashion outfits involves deep understanding of fashion standards while incorporating creativity for choosing multiple fashion items (e.g., Jewelry, Bag, Pants, Dress). In fashion websites, popular or high-quality fashion outfits are usually designed by fashion experts and followed by large audiences. In this paper, we propose a machine learning system to compose fashion outfits automatically. The core of the proposed automatic composition system is to score fashion outfit candidates based on the appearances and meta-data. We propose to leverage outfit popularity on fashion oriented websites to supervise the scoring component. The scoring component is a multi-modal multi-instance deep learning system that evaluates instance aesthetics and set compatibility simultaneously. In order to train and evaluate the proposed composition system, we have collected a large scale fashion outfit dataset with 195K outfits and 368K fashion items from Polyvore. Although the fashion outfit scoring and composition is rather challenging, we have achieved an AUC of 85% for the scoring component, and an accuracy of 77% for a constrained composition task.

IEEE TMM

References in corpus (3)

• Describing Multimedia Content using Attention-based Encoder--Decoder Networks
• Who are the Devils Wearing Prada in New York City?
• User-Curated Image Collections: Modeling and Recommendation

Cited by in corpus (12)

• Explainable Outfit Recommendation with Joint Outfit Matching and Comment Generation
• Dressing as a Whole: Outfit Compatibility Learning Based on Node-wise Graph Neural Networks
• Improving Outfit Recommendation with Co-supervision of Fashion Generation
• COutfitGAN: Learning to Synthesize Compatible Outfits Supervised by Silhouette Masks and Fashion Styles
• Fine-Grained Fashion Similarity Prediction by Attribute-Specific Embedding Learning
• Outfit Compatibility Prediction and Diagnosis with Multi-Layered Comparison Network
• CausalRec: Causal Inference for Visual Debiasing in Visually-Aware Recommendation
• BOXREC: Recommending a Box of Preferred Outfits in Online Shopping
• Sentiment Analysis of Fashion Related Posts in Social Media
• Exploring AI in Fashion: A Review of Aesthetics, Personalization, Virtual Try-On, and Forecasting
• Visual analytics of set data for knowledge discovery and member selection support
• Main Product Detection with Graph Networks for Fashion