A Worrying Reproducibility Study of Intent-Aware Recommendation Models

Abstract

Lately, we have observed a growing interest in intent-aware recommender systems (IARS). The promise of such systems is that they are capable of generating better recommendations by predicting and considering the underlying motivations and short-term goals of consumers. From a technical perspective, various sophisticated neural models were recently proposed in this emerging and promising area. In the broader context of complex neural recommendation models, a growing number of research works unfortunately indicates that (i) reproducing such works is often difficult and (ii) that the true benefits of such models may be limited in reality, e.g., because the reported improvements were obtained through comparisons with untuned or weak baselines. In this work, we investigate if recent research in IARS is similarly affected by such problems. Specifically, we tried to reproduce five contemporary IARS models that were published in top-level outlets, and we benchmarked them against a number of traditional non-neural recommendation models. In two of the cases, running the provided code with the optimal hyperparameters reported in the paper did not yield the results reported in the paper. Worryingly, we find that all examined IARS approaches are consistently outperformed by at least one traditional model. These findings point to sustained methodological issues and to a pressing need for more rigorous scholarly practices.

References in corpus (17)

• Neural Graph Collaborative Filtering
• Disentangled Graph Collaborative Filtering
• Learning Intents behind Interactions with Knowledge Graph for Recommendation
• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• Embarrassingly Shallow Autoencoders for Sparse Data
• A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research
• Disentangled Contrastive Collaborative Filtering
• Empirical Analysis of Session-Based Recommendation Algorithms
• Session-aware Recommendation: A Surprising Quest for the State-of-the-art
• Top-N Recommendation Algorithms: A Quest for the State-of-the-Art
• From Clicks to Carbon: The Environmental Toll of Recommender Systems
• A Survey on Intent-aware Recommender Systems
• The Effect of Third Party Implementations on Reproducibility
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis
• Intent Disentanglement and Feature Self-supervision for Novel Recommendation
• Performance Comparison of Session-based Recommendation Algorithms based on GNNs

Cited by in corpus (1)

• A Multistakeholder Approach to Value-Driven Co-Design of Recommender System Evaluation Metrics in Digital Archives