Formalizing Multimedia Recommendation through Multimodal Deep Learning

Abstract

Recommender systems (RSs) offer personalized navigation experiences on online platforms, but recommendation remains a challenging task, particularly in specific scenarios and domains. Multimodality can help tap into richer information sources and construct more refined user/item profiles for recommendations. However, existing literature lacks a shared and universal schema for modeling and solving the recommendation problem through the lens of multimodality. This work aims to formalize a general multimodal schema for multimedia recommendation. It provides a comprehensive literature review of multimodal approaches for multimedia recommendation from the last eight years, outlines the theoretical foundations of a multimodal pipeline, and demonstrates its rationale by applying it to selected state-of-the-art approaches. The work also conducts a benchmarking analysis of recent algorithms for multimedia recommendation within Elliot, a rigorous framework for evaluating recommender systems. The main aim is to provide guidelines for designing and implementing the next generation of multimodal approaches in multimedia recommendation.

arXiv commentAccepted in the Special Issue on Knowledge Transferring for Recommender Systems (KT4Rec) in ACM Transactions on Recommender Systems

References in corpus (26)

• Neural Graph Collaborative Filtering
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• Self-supervised Graph Learning for Recommendation
• Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering
• Explainable Recommendation: A Survey and New Perspectives
• Mining Latent Structures for Multimedia Recommendation
• Learning Fashion Compatibility with Bidirectional LSTMs
• Multimodal End-to-End Autonomous Driving
• Bootstrap Latent Representations for Multi-modal Recommendation
• MMALFM: Explainable Recommendation by Leveraging Reviews and Images
• A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation
• Multi-Modal Self-Supervised Learning for Recommendation
• Multi-View Graph Convolutional Network for Multimedia Recommendation
• Connecting User and Item Perspectives in Popularity Debiasing for Collaborative Recommendation
• GraphMFT: A Graph Network based Multimodal Fusion Technique for Emotion Recognition in Conversation
• User Diverse Preference Modeling by Multimodal Attentive Metric Learning
• Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation
• M${}^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data
• A Deep Multimodal Approach for Cold-start Music Recommendation
• Market2Dish: Health-aware Food Recommendation
• MEGCF: Multimodal Entity Graph Collaborative Filtering for Personalized Recommendation
• Yum-me: A Personalized Nutrient-based Meal Recommender System
• PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music
• Multimodal Multi-Head Convolutional Attention with Various Kernel Sizes for Medical Image Super-Resolution
• Multimodal Prompting with Missing Modalities for Visual Recognition
• Multimodal Fusion Based Attentive Networks for Sequential Music Recommendation