Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback

Abstract

Probabilistic learning to rank (LTR) has been the dominating approach for optimizing the ranking metric, but cannot maximize long-term rewards. Reinforcement learning models have been proposed to maximize user long-term rewards by formulating the recommendation as a sequential decision-making problem, but could only achieve inferior accuracy compared to LTR counterparts, primarily due to the lack of online interactions and the characteristics of ranking. In this paper, we propose a new off-policy value ranking (VR) algorithm that can simultaneously maximize user long-term rewards and optimize the ranking metric offline for improved sample efficiency in a unified Expectation-Maximization (EM) framework. We theoretically and empirically show that the EM process guides the leaned policy to enjoy the benefit of integration of the future reward and ranking metric, and learn without any online interactions. Extensive offline and online experiments demonstrate the effectiveness of our methods.

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• BPR: Bayesian Personalized Ranking from Implicit Feedback
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Prioritized Experience Replay
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Session-based Recommendations with Recurrent Neural Networks
• Conservative Q-Learning for Offline Reinforcement Learning
• Deep Reinforcement Learning for Page-wise Recommendations
• Reinforcement Learning with Deep Energy-Based Policies
• Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• Safe and Efficient Off-Policy Reinforcement Learning
• Variational Autoencoders for Collaborative Filtering
• Self-Supervised Reinforcement Learning for Recommender Systems