Toward Simulating Environments in Reinforcement Learning Based Recommendations

Abstract

With the recent advances in Reinforcement Learning (RL), there have been tremendous interests in employing RL for recommender systems. However, directly training and evaluating a new RL-based recommendation algorithm needs to collect users' real-time feedback in the real system, which is time and efforts consuming and could negatively impact on users' experiences. Thus, it calls for a user simulator that can mimic real users' behaviors where we can pre-train and evaluate new recommendation algorithms. Simulating users' behaviors in a dynamic system faces immense challenges -- (i) the underlining item distribution is complex, and (ii) historical logs for each user are limited. In this paper, we develop a user simulator base on Generative Adversarial Network (GAN). To be specific, the generator captures the underlining distribution of users' historical logs and generates realistic logs that can be considered as augmentations of real logs; while the discriminator not only distinguishes real and fake logs but also predicts users' behaviors. The experimental results based on real-world e-commerce data demonstrate the effectiveness of the proposed simulator.

References in corpus (12)

• Playing Atari with Deep Reinforcement Learning
• End to End Learning for Self-Driving Cars
• Session-based Recommendations with Recurrent Neural Networks
• Semantic Segmentation using Adversarial Networks
• Deep Reinforcement Learning for Page-wise Recommendations
• Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
• Deep Reinforcement Learning for List-wise Recommendations
• Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modeling
• Deep reinforcement learning for search, recommendation, and online advertising: a survey
• Reinforcement Learning based Recommender System using Biclustering Technique
• Top-K Off-Policy Correction for a REINFORCE Recommender System
• Large-scale Interactive Recommendation with Tree-structured Policy Gradient

Cited by in corpus (8)

• Advances and Challenges in Conversational Recommender Systems: A Survey
• Jointly Learning to Recommend and Advertise
• RecSim: A Configurable Simulation Platform for Recommender Systems
• A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions
• RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems
• AutoLoss: Automated Loss Function Search in Recommendations
• Position Paper on Simulating Privacy Dynamics in Recommender Systems
• MBCAL: Sample Efficient and Variance Reduced Reinforcement Learning for Recommender Systems