Formalizing Multimedia Recommendation through Multimodal Deep Learning

Abstract

Recommender systems (RSs) offer personalized navigation experiences on online platforms, but recommendation remains a challenging task, particularly in specific scenarios and domains. Multimodality can help tap into richer information sources and construct more refined user/item profiles for recommendations. However, existing literature lacks a shared and universal schema for modeling and solving the recommendation problem through the lens of multimodality. This work aims to formalize a general multimodal schema for multimedia recommendation. It provides a comprehensive literature review of multimodal approaches for multimedia recommendation from the last eight years, outlines the theoretical foundations of a multimodal pipeline, and demonstrates its rationale by applying it to selected state-of-the-art approaches. The work also conducts a benchmarking analysis of recent algorithms for multimedia recommendation within Elliot, a rigorous framework for evaluating recommender systems. The main aim is to provide guidelines for designing and implementing the next generation of multimodal approaches in multimedia recommendation.

Accepted in the Special Issue on Knowledge Transferring for Recommender Systems (KT4Rec) in ACM Transactions on Recommender Systems

References in corpus (9)

• Learning Fashion Compatibility with Bidirectional LSTMs
• Multi-View Graph Convolutional Network for Multimedia Recommendation
• Connecting User and Item Perspectives in Popularity Debiasing for Collaborative Recommendation
• M${}^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data
• A Deep Multimodal Approach for Cold-start Music Recommendation
• Market2Dish: Health-aware Food Recommendation
• MEGCF: Multimodal Entity Graph Collaborative Filtering for Personalized Recommendation
• Multimodal Prompting with Missing Modalities for Visual Recognition
• Multimodal Fusion Based Attentive Networks for Sequential Music Recommendation