Generative Slate Recommendation with Reinforcement Learning

Abstract

Recent research has employed reinforcement learning (RL) algorithms to optimize long-term user engagement in recommender systems, thereby avoiding common pitfalls such as user boredom and filter bubbles. They capture the sequential and interactive nature of recommendations, and thus offer a principled way to deal with long-term rewards and avoid myopic behaviors. However, RL approaches are intractable in the slate recommendation scenario - where a list of items is recommended at each interaction turn - due to the combinatorial action space. In that setting, an action corresponds to a slate that may contain any combination of items. While previous work has proposed well-chosen decompositions of actions so as to ensure tractability, these rely on restrictive and sometimes unrealistic assumptions. Instead, in this work we propose to encode slates in a continuous, low-dimensional latent space learned by a variational auto-encoder. Then, the RL agent selects continuous actions in this latent space, which are ultimately decoded into the corresponding slates. By doing so, we are able to (i) relax assumptions required by previous work, and (ii) improve the quality of the action selection by modeling full slates instead of independent items, in particular by enabling diversity. Our experiments performed on a wide array of simulated environments confirm the effectiveness of our generative modeling of slates over baselines in practical scenarios where the restrictive assumptions underlying the baselines are lifted. Our findings suggest that representation learning using generative models is a promising direction towards generalizable RL-based slate recommendation.

arXiv commentWSDM 2023, 9 pages, 3 figures

References in corpus (9)

• Auto-Encoding Variational Bayes
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Recurrent World Models Facilitate Policy Evolution
• Deep Reinforcement Learning in Large Discrete Action Spaces
• Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions
• Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation
• Deep Reinforcement Learning with Attention for Slate Markov Decision Processes with High-Dimensional States and Actions
• The closed loop between opinion formation and personalised recommendations
• Low-Dimensional State and Action Representation Learning with MDP Homomorphism Metrics

Cited by in corpus (4)

• Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstraction
• Proactive Recommendation with Iterative Preference Guidance
• Impression-Aware Recommender Systems
• Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed