Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis

Abstract

The success of graph neural network-based models (GNNs) has significantly advanced recommender systems by effectively modeling users and items as a bipartite, undirected graph. However, many original graph-based works often adopt results from baseline papers without verifying their validity for the specific configuration under analysis. Our work addresses this issue by focusing on the replicability of results. We present a code that successfully replicates results from six popular and recent graph recommendation models (NGCF, DGCF, LightGCN, SGL, UltraGCN, and GFCF) on three common benchmark datasets (Gowalla, Yelp 2018, and Amazon Book). Additionally, we compare these graph models with traditional collaborative filtering models that historically performed well in offline evaluations. Furthermore, we extend our study to two new datasets (Allrecipes and BookCrossing) that lack established setups in existing literature. As the performance on these datasets differs from the previous benchmarks, we analyze the impact of specific dataset characteristics on recommendation accuracy. By investigating the information flow from users' neighborhoods, we aim to identify which models are influenced by intrinsic features in the dataset structure. The code to reproduce our experiments is available at: https://github.com/sisinflab/Graph-RSs-Reproducibility.

Accepted to RecSys '23 - Reproducility Track

References in corpus (22)

• DeepWalk: Online Learning of Social Representations
• LINE: Large-scale Information Network Embedding
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Neural Graph Collaborative Filtering
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• KGAT: Knowledge Graph Attention Network for Recommendation
• Self-supervised Graph Learning for Recommendation
• Disentangled Graph Collaborative Filtering
• Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning
• Knowledge Graph Contrastive Learning for Recommendation
• Hypergraph Contrastive Collaborative Filtering
• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• Collaborative Memory Network for Recommendation Systems
• Embarrassingly Shallow Autoencoders for Sparse Data
• Contrastive Meta Learning with Behavior Multiplicity for Recommendation
• A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research
• Spectral Collaborative Filtering
• Self-Guided Learning to Denoise for Robust Recommendation
• Investigating Accuracy-Novelty Performance for Graph-based Collaborative Filtering
• Top-N Recommendation Algorithms: A Quest for the State-of-the-Art
• Reenvisioning Collaborative Filtering vs Matrix Factorization
• Graph Collaborative Reasoning

Cited by in corpus (5)

• Fair Augmentation for Graph Collaborative Filtering
• A Worrying Reproducibility Study of Intent-Aware Recommendation Models
• Rolling Forward: Enhancing LightGCN with Causal Graph Convolution for Credit Bond Recommendation
• Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing
• Evaluating and Addressing Fairness Across User Groups in Negative Sampling for Recommender Systems