Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing

Abstract

Graph-based techniques relying on neural networks and embeddings have gained attention as a way to develop Recommender Systems (RS) with several papers on the topic presented at SIGIR 2022 and 2023. Given the importance of ensuring that published research is methodologically sound and reproducible, in this paper we analyze 10 graph-based RS papers, most of which were published at SIGIR 2022, and assess their impact on subsequent work published in SIGIR 2023. Our analysis reveals several critical points that require attention: (i) the prevalence of bad practices, such as erroneous data splits or information leakage between training and testing data, which call into question the validity of the results; (ii) frequent inconsistencies between the provided artifacts (source code and data) and their descriptions in the paper, causing uncertainty about what is actually being evaluated; and (iii) the preference for new or complex baselines that are weaker compared to simpler ones, creating the impression of continuous improvement even when, particularly for the Amazon-Book dataset, the state-of-the-art has significantly worsened. Due to these issues, we are unable to confirm the claims made in most of the papers that we examined and attempted to reproduce.

References in corpus (18)

• KGAT: Knowledge Graph Attention Network for Recommendation
• Knowledge Graph Embedding for Link Prediction: A Comparative Analysis
• Knowledge Graph Contrastive Learning for Recommendation
• Hypergraph Contrastive Collaborative Filtering
• Multi-level Cross-view Contrastive Learning for Knowledge-aware Recommender System
• Disentangled Contrastive Collaborative Filtering
• Graph Transformer for Recommendation
• Self-Guided Learning to Denoise for Robust Recommendation
• Reenvisioning Collaborative Filtering vs Matrix Factorization
• Knowledge-refined Denoising Network for Robust Recommendation
• How to Measure the Reproducibility of System-oriented IR Experiments
• INMO: A Model-Agnostic and Scalable Module for Inductive Collaborative Filtering
• The Effect of Third Party Implementations on Reproducibility
• Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis
• Unify Local and Global Information for Top-$N$ Recommendation
• Collaborative Residual Metric Learning
• Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch