Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology

Abstract

Most practical recommender systems focus on estimating immediate user engagement without considering the long-term effects of recommendations on user behavior. Reinforcement learning (RL) methods offer the potential to optimize recommendations for long-term user engagement. However, since users are often presented with slates of multiple items - which may have interacting effects on user choice - methods are required to deal with the combinatorics of the RL action space. In this work, we address the challenge of making slate-based recommendations to optimize long-term value using RL. Our contributions are three-fold. (i) We develop SLATEQ, a decomposition of value-based temporal-difference and Q-learning that renders RL tractable with slates. Under mild assumptions on user choice behavior, we show that the long-term value (LTV) of a slate can be decomposed into a tractable function of its component item-wise LTVs. (ii) We outline a methodology that leverages existing myopic learning-based recommenders to quickly develop a recommender that handles LTV. (iii) We demonstrate our methods in simulation, and validate the scalability of decomposed TD-learning using SLATEQ in live experiments on YouTube.

Short version to appear IJCAI-2019

References in corpus (12)

• Empirical Analysis of Predictive Algorithms for Collaborative Filtering
• A Survey on Multi-Task Learning
• Deep Reinforcement Learning for Page-wise Recommendations
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Cascading Bandits: Learning to Rank in the Cascade Model
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Reinforcement Learning based Recommender System using Biclustering Technique
• Seq2Slate: Re-ranking and Slate Optimization with RNNs
• Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions
• Beyond Greedy Ranking: Slate Optimization via List-CVAE
• Automatic Representation for Lifetime Value Recommender Systems

Cited by in corpus (10)

• Advances and Challenges in Conversational Recommender Systems: A Survey
• Reinforcement Learning Applications
• Do Offline Metrics Predict Online Performance in Recommender Systems?
• RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems
• Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication
• FINN.no Slates Dataset: A new Sequential Dataset Logging Interactions, allViewed Items and Click Responses/No-Click for Recommender Systems Research
• Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation
• Reinforcement Learning for Strategic Recommendations
• A Load Balanced Recommendation Approach
• Deep Exploration for Recommendation Systems