Identifying Offline Metrics that Predict Online Impact: A Pragmatic Strategy for Real-World Recommender Systems

Abstract

A critical challenge in recommender systems is to establish reliable relationships between offline and online metrics that predict real-world performance. Motivated by recent advances in Pareto front approximation, we introduce a pragmatic strategy for identifying offline metrics that align with online impact. A key advantage of this approach is its ability to simultaneously serve multiple test groups, each with distinct offline performance metrics, in an online experiment controlled by a single model. The method is model-agnostic for systems with a neural network backbone, enabling broad applicability across architectures and domains. We validate the strategy through a large-scale online experiment in the field of session-based recommender systems on the OTTO e-commerce platform. The online experiment identifies significant alignments between offline metrics and real-word click-through rate, post-click conversion rate and units sold. Our strategy provides industry practitioners with a valuable tool for understanding offline-to-online metric relationships and making informed, data-driven decisions.

This work was accepted for publication in the 19th ACM Conference on Recommender Systems (RecSys 2025). The final published version will be available at the ACM Digital Library

References in corpus (11)

• Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
• Measuring the Business Value of Recommender Systems
• Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions
• Widespread Flaws in Offline Evaluation of Recommender Systems
• Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions
• Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce
• Pareto Front Approximation for Multi-Objective Session-Based Recommender Systems
• Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation
• Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenders
• Online and Offline Evaluations of Collaborative Filtering and Content Based Recommender Systems
• Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyond