Exploration and Regularization of the Latent Action Space in Recommendation

Abstract

In recommender systems, reinforcement learning solutions have effectively boosted recommendation performance because of their ability to capture long-term user-system interaction. However, the action space of the recommendation policy is a list of items, which could be extremely large with a dynamic candidate item pool. To overcome this challenge, we propose a hyper-actor and critic learning framework where the policy decomposes the item list generation process into a hyper-action inference step and an effect-action selection step. The first step maps the given state space into a vectorized hyper-action space, and the second step selects the item list based on the hyper-action. In order to regulate the discrepancy between the two action spaces, we design an alignment module along with a kernel mapping function for items to ensure inference accuracy and include a supervision module to stabilize the learning process. We build simulated environments on public datasets and empirically show that our framework is superior in recommendation compared to standard RL baselines.

arXiv commentProceedings of the ACM Web Conference 2023 (WWW '23), May 1--5, 2023, Austin, TX, USA

References in corpus (18)

• Addressing Function Approximation Error in Actor-Critic Methods
• Session-based Recommendation with Graph Neural Networks
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Reinforcement Knowledge Graph Reasoning for Explainable Recommendation
• Deep Reinforcement Learning for Page-wise Recommendations
• Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
• Deep Reinforcement Learning in Large Discrete Action Spaces
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• Towards Long-term Fairness in Recommendation
• A Minimalist Approach to Offline Reinforcement Learning
• KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos
• Neural Collaborative Reasoning
• Toward Pareto Efficient Fairness-Utility Trade-off inRecommendation through Reinforcement Learning
• Whole-Chain Recommendations
• Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation
• Counterfactual Collaborative Reasoning
• AutoLossGen: Automatic Loss Function Generation for Recommender Systems
• Constrained Reinforcement Learning for Short Video Recommendation

Cited by in corpus (13)

• LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• Generative Flow Network for Listwise Recommendation
• Explicitly Integrating Judgment Prediction with Legal Document Retrieval: A Law-Guided Generative Approach
• Efficient and Robust Regularized Federated Recommendation
• Future Impact Decomposition in Request-level Recommendations
• Modeling User Retention through Generative Flow Networks
• Prompt Tuning as User Inherent Profile Inference Machine
• Value Function Decomposition in Markov Recommendation Process
• Empowering Denoising Sequential Recommendation with Large Language Model Embeddings
• LLM-Enhanced Reinforcement Learning for Long-Term User Satisfaction in Interactive Recommendation
• SPARK: Adaptive Low-Rank Knowledge Graph Modeling in Hybrid Geometric Spaces for Recommendation
• Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed