Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations
arXiv:2408.12008 · doi:10.1145/3640457.3688195
Abstract
Sequential recommender systems are an important and demanded area of research. Such systems aim to use the order of interactions in a user's history to predict future interactions. The premise is that the order of interactions and sequential patterns play an essential role. Therefore, it is crucial to use datasets that exhibit a sequential structure to evaluate sequential recommenders properly. We apply several methods based on the random shuffling of the user's sequence of interactions to assess the strength of sequential structure across 15 datasets, frequently used for sequential recommender systems evaluation in recent research papers presented at top-tier conferences. As shuffling explicitly breaks sequential dependencies inherent in datasets, we estimate the strength of sequential patterns by comparing metrics for shuffled and original versions of the dataset. Our findings show that several popular datasets have a rather weak sequential structure.
References in corpus (7)
- A Critical Study on Data Leakage in Recommender System Offline Evaluation
- Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
- Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?
- Take a Fresh Look at Recommender Systems from an Evaluation Standpoint
- Rank List Sensitivity of Recommender Systems to Interaction Perturbations
- From Variability to Stability: Advancing RecSys Benchmarking Practices
- Investigating the Robustness of Sequential Recommender Systems Against Training Data Perturbations
Cited by in corpus (5)
- A Survey on Intent-aware Recommender Systems
- Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
- RePlay: a Recommendation Framework for Experimentation and Production Use
- Autoregressive Generation Strategies for Top-K Sequential Recommendations
- Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation