Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems

Abstract

Recommender System (RS) is an important online application that affects billions of users every day. The mainstream RS ranking framework is composed of two parts: a Multi-Task Learning model (MTL) that predicts various user feedback, i.e., clicks, likes, sharings, and a Multi-Task Fusion model (MTF) that combines the multi-task outputs into one final ranking score with respect to user satisfaction. There has not been much research on the fusion model while it has great impact on the final recommendation as the last crucial process of the ranking. To optimize long-term user satisfaction rather than obtain instant returns greedily, we formulate MTF task as Markov Decision Process (MDP) within a recommendation session and propose a Batch Reinforcement Learning (RL) based Multi-Task Fusion framework (BatchRL-MTF) that includes a Batch RL framework and an online exploration. The former exploits Batch RL to learn an optimal recommendation policy from the fixed batch data offline for long-term user satisfaction, while the latter explores potential high-value actions online to break through the local optimal dilemma. With a comprehensive investigation on user behaviors, we model the user satisfaction reward with subtle heuristics from two aspects of user stickiness and user activeness. Finally, we conduct extensive experiments on a billion-sample level real-world dataset to show the effectiveness of our model. We propose a conservative offline policy estimator (Conservative-OPEstimator) to test our model offline. Furthermore, we take online experiments in a real recommendation environment to compare performance of different models. As one of few Batch RL researches applied in MTF task successfully, our model has also been deployed on a large-scale industrial short video platform, serving hundreds of millions of users.

arXiv commentThis paper has been accepted by SIGKDD 2022

References in corpus (10)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Conservative Q-Learning for Offline Reinforcement Learning
• Deep Reinforcement Learning for Page-wise Recommendations
• Off-Policy Deep Reinforcement Learning without Exploration
• Deep Reinforcement Learning in Large Discrete Action Spaces
• Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Deep Spatio-Temporal Neural Networks for Click-Through Rate Prediction
• Value-aware Recommendation based on Reinforced Profit Maximization in E-commerce Systems

Cited by in corpus (11)

• A Survey on Reinforcement Learning for Recommender Systems
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• Multi-Task Recommendations with Reinforcement Learning
• Sequential Recommendation for Optimizing Both Immediate Feedback and Long-term Retention
• LabelCraft: Empowering Short Video Recommendations with Automated Label Crafting
• AutoAssign+: Automatic Shared Embedding Assignment in Streaming Recommendation
• Modeling User Retention through Generative Flow Networks
• AURO: Reinforcement Learning for Adaptive User Retention Optimization in Recommender Systems
• On Gradient Boosted Decision Trees and Neural Rankers: A Case-Study on Short-Video Recommendations at ShareChat
• SPARK: Adaptive Low-Rank Knowledge Graph Modeling in Hybrid Geometric Spaces for Recommendation
• Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed