On Sampling Top-K Recommendation Evaluation

Abstract

Recently, Rendle has warned that the use of sampling-based top-$k$ metrics might not suffice. This throws a number of recent studies on deep learning-based recommendation algorithms, and classic non-deep-learning algorithms using such a metric, into jeopardy. In this work, we thoroughly investigate the relationship between the sampling and global top-$K$ Hit-Ratio (HR, or Recall), originally proposed by Koren[2] and extensively used by others. By formulating the problem of aligning sampling top-$k$ ($SHR@k$) and global top-$K$ ($HR@K$) Hit-Ratios through a mapping function $f$, so that $SHR@k\approx HR@f(k)$, we demonstrate both theoretically and experimentally that the sampling top-$k$ Hit-Ratio provides an accurate approximation of its global (exact) counterpart, and can consistently predict the correct winners (the same as indicate by their corresponding global Hit-Ratios).

References in corpus (7)

• Deep Learning based Recommender System: A Survey and New Perspectives
• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• Collaborative Memory Network for Recommendation Systems
• Embarrassingly Shallow Autoencoders for Sparse Data
• Neural Collaborative Filtering
• On the Difficulty of Evaluating Baselines: A Study on Recommender Systems
• Evaluation Metrics for Item Recommendation under Sampling

Cited by in corpus (7)

• A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models
• Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
• Gradients as an Action: Towards Communication-Efficient Federated Recommender Systems via Adaptive Action Sharing
• Breaking the Top-$K$ Barrier: Advancing Top-$K$ Ranking Metrics Optimization in Recommender Systems
• On Estimating Recommendation Evaluation Metrics under Sampling
• Empowering Denoising Sequential Recommendation with Large Language Model Embeddings
• On the Reliability of Sampling Strategies in Offline Recommender Evaluation