A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research

Abstract

The design of algorithms that generate personalized ranked item lists is a central topic of research in the field of recommender systems. In the past few years, in particular, approaches based on deep learning (neural) techniques have become dominant in the literature. For all of them, substantial progress over the state-of-the-art is claimed. However, indications exist of certain problems in today's research practice, e.g., with respect to the choice and optimization of the baselines used for comparison, raising questions about the published claims. In order to obtain a better understanding of the actual progress, we have tried to reproduce recent results in the area of neural recommendation approaches based on collaborative filtering. The worrying outcome of the analysis of these recent works-all were published at prestigious scientific conferences between 2015 and 2018-is that 11 out of the 12 reproducible neural approaches can be outperformed by conceptually simple methods, e.g., based on the nearest-neighbor heuristics. None of the computationally complex neural methods was actually consistently better than already existing learning-based techniques, e.g., using matrix factorization or linear models. In our analysis, we discuss common issues in today's research practice, which, despite the many papers that are published on the topic, have apparently led the field to a certain level of stagnation.

arXiv commentSource code and full results available at: https://github.com/MaurizioFD/RecSys2019_DeepLearning_Evaluation

References in corpus (11)

• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• Evaluation of Session-based Recommendation Algorithms
• Collaborative Memory Network for Recommendation Systems
• Embarrassingly Shallow Autoencoders for Sparse Data
• Multi-Pointer Co-Attention Networks for Recommendation
• Latent Relational Metric Learning via Memory-based Attention for Collaborative Ranking
• Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation
• Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models
• Attentive Neural Architecture Incorporating Song Features For Music Recommendation
• Artist-driven layering and user's behaviour impact on recommendations in a playlist continuation scenario
• Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems

Cited by in corpus (39)

• A Survey on Accuracy-oriented Neural Recommendation: From Collaborative Filtering to Information-rich Recommendation
• A Review of Biomedical Datasets Relating to Drug Discovery: A Knowledge Graph Perspective
• A Survey on Popularity Bias in Recommender Systems
• Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
• Feature Selection for Recommender Systems with Quantum Computing
• Take a Fresh Look at Recommender Systems from an Evaluation Standpoint
• Reenvisioning Collaborative Filtering vs Matrix Factorization
• The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
• A Survey on Point-of-Interest Recommendations Leveraging Heterogeneous Data
• Towards a Better Understanding of Linear Models for Recommendation
• GAN-based Matrix Factorization for Recommender Systems
• Random Walks with Erasure: Diversifying Personalized Recommendations on Social and Information Networks
• The Effect of Third Party Implementations on Reproducibility
• Economic Recommender Systems -- A Systematic Review
• The complementarity of a diverse range of deep learning features extracted from video content for video recommendation
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis
• A Methodology for the Offline Evaluation of Recommender Systems in a User Interface with Multiple Carousels
• A Reproducible Analysis of Sequential Recommender Systems
• Recommender Systems for Good (RS4Good): Survey of Use Cases and a Call to Action for Research that Matters
• From Variability to Stability: Advancing RecSys Benchmarking Practices
• Measuring the User Satisfaction in a Recommendation Interface with Multiple Carousels
• Learning Recommendations from User Actions in the Item-poor Insurance Domain
• Performance Comparison of Session-based Recommendation Algorithms based on GNNs
• Sensitivity of low-rank matrix recovery
• Impacts of Mainstream-Driven Algorithms on Recommendations for Children Across Domains: A Reproducibility Study
• A Worrying Reproducibility Study of Intent-Aware Recommendation Models
• Analyzing the Effectiveness of Quantum Annealing with Meta-Learning
• Fuzzy Norm-Explicit Product Quantization for Recommender Systems
• Eye Tracking as a Source of Implicit Feedback in Recommender Systems: A Preliminary Analysis
• An Evaluation Study of Generative Adversarial Networks for Collaborative Filtering
• Beyond Collaborative Filtering: A Relook at Task Formulation in Recommender Systems
• It's Enough: Relaxing Diagonal Constraints in Linear Autoencoders for Recommendation
• Recommending Target Actions Outside Sessions in the Data-poor Insurance Domain
• Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing
• Impression-Aware Recommender Systems
• The Hidden Cost of Defaults in Recommender System Evaluation
• Can Offline Metrics Measure Explanation Goals? A Comparative Survey Analysis of Offline Explanation Metrics in Recommender Systems
• Searching Personal Collections
• Point of Interest Recommendation: Pitfalls and Viable Solutions