On the discriminative power of Hyper-parameters in Cross-Validation and how to choose them
arXiv:1909.02523 · doi:10.1145/3298689.3347010
Abstract
Hyper-parameters tuning is a crucial task to make a model perform at its best. However, despite the well-established methodologies, some aspects of the tuning remain unexplored. As an example, it may affect not just accuracy but also novelty as well as it may depend on the adopted dataset. Moreover, sometimes it could be sufficient to concentrate on a single parameter only (or a few of them) instead of their overall set. In this paper we report on our investigation on hyper-parameters tuning by performing an extensive 10-Folds Cross-Validation on MovieLens and Amazon Movies for three well-known baselines: User-kNN, Item-kNN, BPR-MF. We adopted a grid search strategy considering approximately 15 values for each parameter, and we then evaluated each combination of parameters in terms of accuracy and novelty. We investigated the discriminative power of nDCG, Precision, Recall, MRR, EFD, EPC, and, finally, we analyzed the role of parameters on model evaluation for Cross-Validation.
arXiv comment5 pages RecSys 2019
References in corpus (4)
- Practical Bayesian Optimization of Machine Learning Algorithms
- BPR: Bayesian Personalized Ranking from Implicit Feedback
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Recommending Learning Algorithms and Their Associated Hyperparameters
Cited by in corpus (8)
- Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation
- Top-N Recommendation Algorithms: A Quest for the State-of-the-Art
- Search Algorithms for Automated Hyper-Parameter Tuning
- How to Put Users in Control of their Data in Federated Top-N Recommendation with Learning to Rank
- Understanding the Influence of Data Characteristics on the Performance of Point-of-Interest Recommendation Algorithms
- Simulations for novel problems in recommendation: analyzing misinformation and data characteristics
- Do We Really Need Specialization? Evaluating Generalist Text Embeddings for Zero-Shot Recommendation and Search
- Explaining the Performance of Collaborative Filtering Methods With Optimal Data Characteristics