Dataset and Models for Item Recommendation Using Multi-Modal User Interactions

Abstract

While recommender systems with multi-modal item representations (image, audio, and text), have been widely explored, learning recommendations from multi-modal user interactions (e.g., clicks and speech) remains an open problem. We study the case of multi-modal user interactions in a setting where users engage with a service provider through multiple channels (website and call center). In such cases, incomplete modalities naturally occur, since not all users interact through all the available channels. To address these challenges, we publish a real-world dataset that allows progress in this under-researched area. We further present and benchmark various methods for leveraging multi-modal user interactions for item recommendations, and propose a novel approach that specifically deals with missing modalities by mapping user interactions to a common feature space. Our analysis reveals important interactions between the different modalities and that a frequently occurring modality can enhance learning from a less frequent one.

References in corpus (4)

• Multi-Behavior Enhanced Recommendation with Cross-Interaction Collaborative Relation Modeling
• Multi-behavior Self-supervised Learning for Recommendation
• Learning Recommendations from User Actions in the Item-poor Insurance Domain
• Recommending Target Actions Outside Sessions in the Data-poor Insurance Domain