Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches

Abstract

Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. With the strongly increased interest in machine learning in general, it has, as a result, become difficult to keep track of what represents the state-of-the-art at the moment, e.g., for top-n recommendation tasks. At the same time, several recent publications point out problems in today's research practice in applied machine learning, e.g., in terms of the reproducibility of the results or the choice of the baselines when proposing new models. In this work, we report the results of a systematic analysis of algorithmic proposals for top-n recommendation tasks. Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. The remaining one clearly outperformed the baselines but did not consistently outperform a well-tuned non-neural linear ranking method. Overall, our work sheds light on a number of potential problems in today's machine learning scholarship and calls for improved scientific practices in this area. Source code of our experiments and full results are available at: https://github.com/MaurizioFD/RecSys2019_DeepLearning_Evaluation.

arXiv commentSource code available at: https://github.com/MaurizioFD/RecSys2019_DeepLearning_Evaluation

References in corpus (8)

• Evaluation of Session-based Recommendation Algorithms
• Collaborative Memory Network for Recommendation Systems
• Multi-Pointer Co-Attention Networks for Recommendation
• Latent Relational Metric Learning via Memory-based Attention for Collaborative Ranking
• Spectral Collaborative Filtering
• Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation
• Attentive Neural Architecture Incorporating Song Features For Music Recommendation
• Artist-driven layering and user's behaviour impact on recommendations in a playlist continuation scenario

Cited by in corpus (81)

• Big-Data Science in Porous Materials: Materials Genomics and Machine Learning
• DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems
• A Survey on Accuracy-oriented Neural Recommendation: From Collaborative Filtering to Information-rich Recommendation
• Measuring the Business Value of Recommender Systems
• User-oriented Fairness in Recommendation
• A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research
• Empirical Analysis of Session-Based Recommendation Algorithms
• MetaKG: Meta-learning on Knowledge Graph for Cold-start Recommendation
• Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation
• CARCA: Context and Attribute-Aware Next-Item Recommendation via Cross-Attention
• Neural Collaborative Reasoning
• Analyzing Item Popularity Bias of Music Recommender Systems: Are Different Genders Equally Affected?
• How Useful are Reviews for Recommendation? A Critical Review and Potential Improvements
• Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
• A Re-visit of the Popularity Baseline in Recommender Systems
• Feature Selection for Recommender Systems with Quantum Computing
• Unlearning Protected User Attributes in Recommendations with Adversarial Training
• On Sampling Top-K Recommendation Evaluation
• Recommender systems based on graph embedding techniques: A comprehensive review
• Take a Fresh Look at Recommender Systems from an Evaluation Standpoint
• Recommendations and User Agency: The Reachability of Collaboratively-Filtered Information
• Representation Learning and Pairwise Ranking for Implicit Feedback in Recommendation Systems
• Space4HGNN: A Novel, Modularized and Reproducible Platform to Evaluate Heterogeneous Graph Neural Network
• Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
• Performance of Hyperbolic Geometry Models on Top-N Recommendation Tasks
• Consumer Fairness in Recommender Systems: Contextualizing Definitions and Mitigations
• How to Measure the Reproducibility of System-oriented IR Experiments
• Widespread Flaws in Offline Evaluation of Recommender Systems
• Mutually-Regularized Dual Collaborative Variational Auto-encoder for Recommendation Systems
• Benchmarking Network Embedding Models for Link Prediction: Are We Making Progress?
• DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation
• GraphWorld: Fake Graphs Bring Real Insights for GNNs
• Towards a Better Understanding of Linear Models for Recommendation
• ContentWise Impressions: An Industrial Dataset with Impressions Included
• Reproducibility, Replicability and Beyond: Assessing Production Readiness of Aspect Based Sentiment Analysis in the Wild
• GAN-based Matrix Factorization for Recommender Systems
• Random Walks with Erasure: Diversifying Personalized Recommendations on Social and Information Networks
• Economic Recommender Systems -- A Systematic Review
• The Effect of Third Party Implementations on Reproducibility
• Local Collaborative Autoencoders
• Masked and Swapped Sequence Modeling for Next Novel Basket Recommendation in Grocery Shopping
• Looking at CTR Prediction Again: Is Attention All You Need?
• All for One, and One for All: UrbanSyn Dataset, the third Musketeer of Synthetic Driving Scenes
• The complementarity of a diverse range of deep learning features extracted from video content for video recommendation
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline
• Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis
• A Reproducible Analysis of Sequential Recommender Systems
• Yes We Care! -- Certification for Machine Learning Methods through the Care Label Framework
• Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
• A Scalable Framework for Automatic Playlist Continuation on Music Streaming Services
• Toward a Better Understanding of Loss Functions for Collaborative Filtering
• Curse of "Low" Dimensionality in Recommender Systems
• Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems
• Fair Augmentation for Graph Collaborative Filtering
• The Importance of Causality in Decision Making: A Perspective on Recommender Systems
• Capturing Popularity Trends: A Simplistic Non-Personalized Approach for Enhanced Item Recommendation
• Efficient Model-Based Collaborative Filtering with Fast Adaptive PCA
• Performance Comparison of Session-based Recommendation Algorithms based on GNNs
• Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering
• A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems
• Cross-Domain Latent Factors Sharing via Implicit Matrix Factorization
• Towards Multi-Language Recipe Personalisation and Recommendation
• EvalNE: A Framework for Evaluating Network Embeddings on Link Prediction
• A Worrying Reproducibility Study of Intent-Aware Recommendation Models
• Understanding the Influence of Data Characteristics on the Performance of Point-of-Interest Recommendation Algorithms
• D-RDW: Diversity-Driven Random Walks for News Recommender Systems
• It's Enough: Relaxing Diagonal Constraints in Linear Autoencoders for Recommendation
• Beyond Collaborative Filtering: A Relook at Task Formulation in Recommender Systems
• An Evaluation Study of Generative Adversarial Networks for Collaborative Filtering
• Efficient Hyperparameter Optimization under Multi-Source Covariate Shift
• Recommendation Is a Dish Better Served Warm
• Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing
• The Hidden Cost of Defaults in Recommender System Evaluation
• When the Label Ignores the Request: Auditing Policy-Selected Targets in Synthetic Conversational Music Recommendation
• Does Weighting Improve Matrix Factorization for Recommender Systems?
• APS Explorer: Navigating Algorithm Performance Spaces for Informed Dataset Selection
• NU:BRIEF -- A Privacy-aware Newsletter Personalization Engine for Publishers
• ZoRRO: A Zero-Weight Personalized Recommender System for Scalable News Recommendation
• Collaborative Filtering Through Weighted Similarities of User and Item Embeddings
• Lusifer: LLM-based User SImulated Feedback Environment for online Recommender systems