Exposure-Based Reinforcement Learning to Rank

Abstract

Reinforcement learning (RL) methods for learning-to-rank (LTR) can optimize (almost) any ranking goal, e.g., from precision or discounted cumulative gain to fairness-of-exposure or ranking distillation. However, standard RL is ineffective and computationally costly due to the enormous action space in LTR settings. Existing methods reach computational efficiency through custom gradient computation algorithms, but they are very complex to implement and often clash with auto-differentiation. Consequently, existing RL for LTR is not attractive to many practitioners. We reconsider RL for LTR while actively avoiding reliance on custom gradients. Contrary to the existing approaches, we focus on variance reduction and GPU computation. In doing so, we discover that high sample-efficiency can be reached through baseline corrections and partial marginalization. Furthermore, we propose an abstraction that places gradient estimation behind a document-exposure distribution, this enables seamless plug-and-play integration with auto-differentiation. Thereby, one only has to implement a loss as a differentiable function of exposure and RL for LTR can optimize it using auto-differentiation. Our experimental results reveal that our new exposure-based RL for LTR approach converges considerably faster and at significantly higher ranking performance than existing custom gradients, with no additional costs in computation time when using GPUs. In contrast, existing custom gradients result in severe stability issues when converging over many epochs, which never occur for our methods. Thus, we considerably improve RL for LTR methodology by increasing its effectiveness, efficiency, and ease of application.

Published at ICTIR'26

References in corpus (10)

• Fairness of Exposure in Rankings
• Equity of Attention: Amortizing Individual Fairness in Rankings
• Evaluating Stochastic Rankings with Expected Exposure
• TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank
• Differentiable Unbiased Online Learning to Rank
• Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank
• Ranking for Relevance and Display Preferences in Complex Presentation Layouts
• Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking
• Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity
• Optimal Baseline Corrections for Off-Policy Contextual Bandits