Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation

Abstract

Recommender Systems have shown to be an effective way to alleviate the over-choice problem and provide accurate and tailored recommendations. However, the impressive number of proposed recommendation algorithms, splitting strategies, evaluation protocols, metrics, and tasks, has made rigorous experimental evaluation particularly challenging. Puzzled and frustrated by the continuous recreation of appropriate evaluation benchmarks, experimental pipelines, hyperparameter optimization, and evaluation procedures, we have developed an exhaustive framework to address such needs. Elliot is a comprehensive recommendation framework that aims to run and reproduce an entire experimental pipeline by processing a simple configuration file. The framework loads, filters, and splits the data considering a vast set of strategies (13 splitting methods and 8 filtering approaches, from temporal training-test splitting to nested K-folds Cross-Validation). Elliot optimizes hyperparameters (51 strategies) for several recommendation algorithms (50), selects the best models, compares them with the baselines providing intra-model statistics, computes metrics (36) spanning from accuracy to beyond-accuracy, bias, and fairness, and conducts statistical analysis (Wilcoxon and Paired t-test). The aim is to provide the researchers with a tool to ease (and make them reproducible) all the experimental evaluation phases, from data reading to results collection. Elliot is available on GitHub (https://github.com/sisinflab/elliot).

Published at SIGIR 2021, 10 pages, 1 figure

Cited by in corpus (14)

• Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
• Top-N Recommendation Algorithms: A Quest for the State-of-the-Art
• From Clicks to Carbon: The Environmental Toll of Recommender Systems
• Formalizing Multimedia Recommendation through Multimodal Deep Learning
• DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline
• A Reproducible Analysis of Sequential Recommender Systems
• Black-box Error Diagnosis in Deep Neural Networks for Computer Vision: a Survey of Tools
• Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
• RePlay: a Recommendation Framework for Experimentation and Production Use
• Informfully Recommenders -- Reproducibility Framework for Diversity-aware Intra-session Recommendations
• Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives
• The Hidden Cost of Defaults in Recommender System Evaluation
• Parameter-Efficient Single Collaborative Branch for Recommendation