Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs

Abstract

Scalability issue plays a crucial role in productionizing modern recommender systems. Even lightweight architectures may suffer from high computational overload due to intermediate calculations, limiting their practicality in real-world applications. Specifically, applying full Cross-Entropy (CE) loss often yields state-of-the-art performance in terms of recommendations quality. Still, it suffers from excessive GPU memory utilization when dealing with large item catalogs. This paper introduces a novel Scalable Cross-Entropy (SCE) loss function in the sequential learning setup. It approximates the CE loss for datasets with large-size catalogs, enhancing both time efficiency and memory usage without compromising recommendations quality. Unlike traditional negative sampling methods, our approach utilizes a selective GPU-efficient computation strategy, focusing on the most informative elements of the catalog, particularly those most likely to be false positives. This is achieved by approximating the softmax distribution over a subset of the model outputs through the maximum inner product search. Experimental results on multiple datasets demonstrate the effectiveness of SCE in reducing peak memory usage by a factor of up to 100 compared to the alternatives, retaining or even exceeding their metrics values. The proposed approach also opens new perspectives for large-scale developments in different domains, such as large language models.

11 pages, fixed some typos

References in corpus (9)

• Vista: A Visually, Socially, and Temporally-aware Model for Artistic Recommendation
• Contrastive Self-supervised Sequential Recommendation with Robust Augmentation
• Denoising Self-attentive Sequential Recommendation
• A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models
• Simplify and Robustify Negative Sampling for Implicit Collaborative Filtering
• Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?
• gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling
• Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions
• RECE: Reduced Cross-Entropy Loss for Large-Catalogue Sequential Recommenders

Cited by in corpus (4)

• Time to Split: Exploring Data Splitting Strategies for Offline Evaluation of Sequential Recommenders
• Recommendation Is a Dish Better Served Warm
• Autoregressive Generation Strategies for Top-K Sequential Recommendations
• eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion