Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation

Abstract

Offline reinforcement learning (RL), a technology that offline learns a policy from logged data without the need to interact with online environments, has become a favorable choice in decision-making processes like interactive recommendation. Offline RL faces the value overestimation problem. To address it, existing methods employ conservatism, e.g., by constraining the learned policy to be close to behavior policies or punishing the rarely visited state-action pairs. However, when applying such offline RL to recommendation, it will cause a severe Matthew effect, i.e., the rich get richer and the poor get poorer, by promoting popular items or categories while suppressing the less popular ones. It is a notorious issue that needs to be addressed in practical recommender systems. In this paper, we aim to alleviate the Matthew effect in offline RL-based recommendation. Through theoretical analyses, we find that the conservatism of existing methods fails in pursuing users' long-term satisfaction. It inspires us to add a penalty term to relax the pessimism on states with high entropy of the logging policy and indirectly penalizes actions leading to less diverse states. This leads to the main technical contribution of the work: Debiased model-based Offline RL (DORL) method. Experiments show that DORL not only captures user interests well but also alleviates the Matthew effect. The implementation is available via https://github.com/chongminggao/DORL-codes.

arXiv commentSIGIR 2023 Full Paper

References in corpus (31)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Off-Policy Deep Reinforcement Learning without Exploration
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• MOPO: Model-based Offline Policy Optimization
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• MOReL : Model-Based Offline Reinforcement Learning
• KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems
• Offline A/B testing for Recommender Systems
• DGCN: Diversified Recommendation with Graph Convolutional Networks
• Offline Reinforcement Learning with Implicit Q-Learning
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
• KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos
• An Optimistic Perspective on Offline Reinforcement Learning
• Critic Regularized Regression
• Toward Pareto Efficient Fairness-Utility Trade-off inRecommendation through Reinforcement Learning
• An Audit of Misinformation Filter Bubbles on YouTube: Bubble Bursting and Recent Behavior Changes
• COMBO: Conservative Offline Model-Based Policy Optimization
• Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems
• Recommending Accurate and Diverse Items Using Bilateral Branch Network
• Exploration and Regularization of the Latent Action Space in Recommendation
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Quantitative analysis of Matthew effect and sparsity problem of recommender systems
• Dynamic Causal Collaborative Filtering
• State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study
• ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor
• Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective
• Offline Evaluation for Reinforcement Learning-based Recommendation: A Critical Issue and Some Alternatives

Cited by in corpus (11)

• Large Language Models are Learnable Planners for Long-Term Recommendation
• SPRec: Self-Play to Debias LLM-based Recommendation
• How Do Recommendation Models Amplify Popularity Bias? An Analysis from the Spectral Perspective
• Future Impact Decomposition in Request-level Recommendations
• Rankformer: A Graph Transformer for Recommendation based on Ranking Objective
• Breaking the Top-$K$ Barrier: Advancing Top-$K$ Ranking Metrics Optimization in Recommender Systems
• A Unified Online-Offline Framework for Co-Branding Campaign Recommendations
• LLM-Enhanced Reinforcement Learning for Long-Term User Satisfaction in Interactive Recommendation
• Towards Two-Stage Counterfactual Learning to Rank
• Not All Videos Become Outdated: Short-Video Recommendation by Learning to Deconfound Release Interval Bias
• Reward Balancing Revisited: Enhancing Offline Reinforcement Learning for Recommender Systems