Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems

Abstract

Recommender systems play a crucial role in our daily lives. Feed streaming mechanism has been widely used in the recommender system, especially on the mobile Apps. The feed streaming setting provides users the interactive manner of recommendation in never-ending feeds. In such an interactive manner, a good recommender system should pay more attention to user stickiness, which is far beyond classical instant metrics, and typically measured by {\bf long-term user engagement}. Directly optimizing the long-term user engagement is a non-trivial problem, as the learning target is usually not available for conventional supervised learning methods. Though reinforcement learning~(RL) naturally fits the problem of maximizing the long term rewards, applying RL to optimize long-term user engagement is still facing challenges: user behaviors are versatile and difficult to model, which typically consists of both instant feedback~(e.g. clicks, ordering) and delayed feedback~(e.g. dwell time, revisit); in addition, performing effective off-policy learning is still immature, especially when combining bootstrapping and function approximation. To address these issues, in this work, we introduce a reinforcement learning framework --- FeedRec to optimize the long-term user engagement. FeedRec includes two components: 1)~a Q-Network which designed in hierarchical LSTM takes charge of modeling complex user behaviors, and 2)~an S-Network, which simulates the environment, assists the Q-Network and voids the instability of convergence in policy learning. Extensive experiments on synthetic data and a real-world large scale data show that FeedRec effectively optimizes the long-term user engagement and outperforms state-of-the-arts.

References in corpus (9)

• Playing Atari with Deep Reinforcement Learning
• Session-based Recommendations with Recurrent Neural Networks
• Deep Reinforcement Learning for Page-wise Recommendations
• Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
• Recommendations as Treatments: Debiasing Learning and Evaluation
• Deep Reinforcement Learning for List-wise Recommendations
• Deep reinforcement learning for search, recommendation, and online advertising: a survey
• Partially Observable Markov Decision Process for Recommender Systems
• Off-policy Learning for Multiple Loggers

Cited by in corpus (5)

• Self-Supervised Reinforcement Learning for Recommender Systems
• Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication
• Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning
• Pre-trained Language Model based Ranking in Baidu Search
• Off-policy Learning for Multiple Loggers