A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems

Abstract

Since the creation of the Web, recommender systems (RSs) have been an indispensable mechanism in information filtering. State-of-the-art RSs primarily depend on categorical features, which ecoded by embedding vectors, resulting in excessively large embedding tables. To prevent over-parameterized embedding tables from harming scalability, both academia and industry have seen increasing efforts in compressing RS embeddings. However, despite the prosperity of lightweight embedding-based RSs (LERSs), a wide diversity is seen in evaluation protocols, resulting in obstacles when relating LERS performance to real-world usability. Moreover, despite the common goal of lightweight embeddings, LERSs are evaluated with a single choice between the two main recommendation tasks -- collaborative filtering and content-based recommendation. This lack of discussions on cross-task transferability hinders the development of unified, more scalable solutions. Motivated by these issues, this study investigates various LERSs' performance, efficiency, and cross-task transferability via a thorough benchmarking process. Additionally, we propose an efficient embedding compression method using magnitude pruning, which is an easy-to-deploy yet highly competitive baseline that outperforms various complex LERSs. Our study reveals the distinct performance of LERSs across the two tasks, shedding light on their effectiveness and generalizability. To support edge-based recommendations, we tested all LERSs on a Raspberry Pi 4, where the efficiency bottleneck is exposed. Finally, we conclude this paper with critical summaries of LERS performance, model selection suggestions, and underexplored challenges around LERSs for future research. To encourage future research, we publish source codes and artifacts at \href{this link}{https://github.com/chenxing1999/recsys-benchmark}.

References in corpus (22)

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Learning both Weights and Connections for Efficient Neural Networks
• DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems
• Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches
• LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Are Graph Augmentations Necessary? Simple Graph Contrastive Learning for Recommendation
• OptEmbed: Learning Optimal Embedding Table for Click-through Rate Prediction
• DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation
• Embedding Compression in Recommender Systems: A Survey
• Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation
• Towards Mitigating Dimensional Collapse of Representations in Collaborative Filtering
• BARS: Towards Open Benchmarking for Recommender Systems
• Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems
• Learning Compressed Embeddings for On-Device Inference
• Continuous Input Embedding Size Search For Recommender Systems
• Lightweight Embeddings for Graph Collaborative Filtering
• Scalable Dynamic Embedding Size Search for Streaming Recommendation
• Model-Agnostic Decentralized Collaborative Learning for On-Device POI Recommendation