The Effect of Third Party Implementations on Reproducibility

Abstract

Reproducibility of recommender systems research has come under scrutiny during recent years. Along with works focusing on repeating experiments with certain algorithms, the research community has also started discussing various aspects of evaluation and how these affect reproducibility. We add a novel angle to this discussion by examining how unofficial third-party implementations could benefit or hinder reproducibility. Besides giving a general overview, we thoroughly examine six third-party implementations of a popular recommender algorithm and compare them to the official version on five public datasets. In the light of our alarming findings we aim to draw the attention of the research community to this neglected aspect of reproducibility.

arXiv commentAppearing in the Proceedings of the 17th ACM Conference on Recommender Systems (RecSys'23)

References in corpus (10)

• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Session-based Recommendation with Graph Neural Networks
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
• Global Context Enhanced Graph Neural Networks for Session-based Recommendation
• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research
• NISER: Normalized Item and Session Representations to Handle Popularity Bias
• BARS: Towards Open Benchmarking for Recommender Systems

Cited by in corpus (9)

• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline
• A Reproducible Analysis of Sequential Recommender Systems
• Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters
• Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
• From Variability to Stability: Advancing RecSys Benchmarking Practices
• Performance Comparison of Session-based Recommendation Algorithms based on GNNs
• A Worrying Reproducibility Study of Intent-Aware Recommendation Models
• Beyond Collaborative Filtering: A Relook at Task Formulation in Recommender Systems
• Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing