Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation

Abstract

Recommender Systems have shown to be an effective way to alleviate the over-choice problem and provide accurate and tailored recommendations. However, the impressive number of proposed recommendation algorithms, splitting strategies, evaluation protocols, metrics, and tasks, has made rigorous experimental evaluation particularly challenging. Puzzled and frustrated by the continuous recreation of appropriate evaluation benchmarks, experimental pipelines, hyperparameter optimization, and evaluation procedures, we have developed an exhaustive framework to address such needs. Elliot is a comprehensive recommendation framework that aims to run and reproduce an entire experimental pipeline by processing a simple configuration file. The framework loads, filters, and splits the data considering a vast set of strategies (13 splitting methods and 8 filtering approaches, from temporal training-test splitting to nested K-folds Cross-Validation). Elliot optimizes hyperparameters (51 strategies) for several recommendation algorithms (50), selects the best models, compares them with the baselines providing intra-model statistics, computes metrics (36) spanning from accuracy to beyond-accuracy, bias, and fairness, and conducts statistical analysis (Wilcoxon and Paired t-test). The aim is to provide the researchers with a tool to ease (and make them reproducible) all the experimental evaluation phases, from data reading to results collection. Elliot is available on GitHub (https://github.com/sisinflab/elliot).

arXiv commentPublished at SIGIR 2021, 10 pages, 1 figure

References in corpus (7)

• Neural Graph Collaborative Filtering
• NAIS: Neural Attentive Item Similarity Model for Recommendation
• IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models
• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• Empirical Analysis of Session-Based Recommendation Algorithms
• Generating Reliable Friends via Adversarial Training to Improve Social Recommendation
• On the discriminative power of Hyper-parameters in Cross-Validation and how to choose them

Cited by in corpus (15)

• A Critical Study on Data Leakage in Recommender System Offline Evaluation
• Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
• Top-N Recommendation Algorithms: A Quest for the State-of-the-Art
• From Clicks to Carbon: The Environmental Toll of Recommender Systems
• Formalizing Multimedia Recommendation through Multimodal Deep Learning
• DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline
• A Reproducible Analysis of Sequential Recommender Systems
• Black-box Error Diagnosis in Deep Neural Networks for Computer Vision: a Survey of Tools
• Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
• RePlay: a Recommendation Framework for Experimentation and Production Use
• Informfully Recommenders -- Reproducibility Framework for Diversity-aware Intra-session Recommendations
• The Hidden Cost of Defaults in Recommender System Evaluation
• Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives
• Parameter-Efficient Single Collaborative Branch for Recommendation